[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:56:24 dist-test-slave-tk9z krb5kdc[25252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:56:24 dist-test-slave-tk9z krb5kdc[25252](info): set up 2 sockets
Aug 08 12:56:24 dist-test-slave-tk9z krb5kdc[25252](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 12:56:25.637257 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.466s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:56:25 dist-test-slave-tk9z krb5kdc[25252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:56:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:56:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 12:56:25.828656 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:46647
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:46647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:26.271641 25268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:26.271997 25268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:26.272158 25268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:26.280632 25268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:56:26.280748 25268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:26.280804 25268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:26.280889 25268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:56:26.280969 25268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:56:26.291862 25268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:46647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:46647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:26.293490 25268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:26.294941 25268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:26.308290 25273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:26.308375 25276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:26.308435 25274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:26.309147 25268 server_base.cc:1034] running on GCE node
I20260808 12:56:26.310283 25268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:26.311900 25268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:26.313150 25268 hybrid_clock.cc:648] HybridClock initialized: now 1786193786313108 us; error 52 us; skew 500 ppm
I20260808 12:56:26.313516 25268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:26 dist-test-slave-tk9z krb5kdc[25252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193786, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:26.317001 25268 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:56:26.318813 25268 webserver.cc:460] Webserver started at http://127.24.167.62:35241/ using document root <none> and password file <none>
I20260808 12:56:26.319710 25268 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:26.319908 25268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:26.320267 25268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:26.323671 25268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "ee8d801f5f354b9da0e0b75d2d4eb108"
format_stamp: "Formatted at 2026-08-08 12:56:26 on dist-test-slave-tk9z"
server_key: "dc8726d7508f61429a000273544e9a83"
server_key_iv: "e5db2d6e831c8de4e522788330db9034"
server_key_version: "encryptionkey@0"
I20260808 12:56:26.324548 25268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "ee8d801f5f354b9da0e0b75d2d4eb108"
format_stamp: "Formatted at 2026-08-08 12:56:26 on dist-test-slave-tk9z"
server_key: "dc8726d7508f61429a000273544e9a83"
server_key_iv: "e5db2d6e831c8de4e522788330db9034"
server_key_version: "encryptionkey@0"
I20260808 12:56:26.330299 25268 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 12:56:26.334468 25283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:26.335806 25268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 12:56:26.336099 25268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "ee8d801f5f354b9da0e0b75d2d4eb108"
format_stamp: "Formatted at 2026-08-08 12:56:26 on dist-test-slave-tk9z"
server_key: "dc8726d7508f61429a000273544e9a83"
server_key_iv: "e5db2d6e831c8de4e522788330db9034"
server_key_version: "encryptionkey@0"
I20260808 12:56:26.336378 25268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:26.359292 25268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:26.365000 25268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:26.365311 25268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:26.395036 25268 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:46647
I20260808 12:56:26.395156 25352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:46647 every 8 connection(s)
I20260808 12:56:26.396736 25268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:56:26.400372 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 25268
I20260808 12:56:26.400854 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:56:26.402523 25353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:26.403872 25244 external_mini_cluster.cc:1428] Setting key f6ad0cfd7aa54b68b02a28597e64b0a9
I20260808 12:56:26.416900 25353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108: Bootstrap starting.
I20260808 12:56:26.420852 25353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:26.422878 25353 log.cc:824] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108: Log is configured to *not* fsync() on all Append() calls
I20260808 12:56:26.426486 25353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108: No bootstrap required, opened a new log
I20260808 12:56:26.436903 25353 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8d801f5f354b9da0e0b75d2d4eb108" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46647 } }
I20260808 12:56:26.437260 25353 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:26.437413 25353 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8d801f5f354b9da0e0b75d2d4eb108, State: Initialized, Role: FOLLOWER
I20260808 12:56:26.438201 25353 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8d801f5f354b9da0e0b75d2d4eb108" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46647 } }
I20260808 12:56:26.438539 25353 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:56:26.438709 25353 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:56:26.438894 25353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [term 0 FOLLOWER]: Advancing to term 1
Aug 08 12:56:26 dist-test-slave-tk9z krb5kdc[25252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:26.443063 25353 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8d801f5f354b9da0e0b75d2d4eb108" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46647 } }
I20260808 12:56:26.443792 25353 leader_election.cc:302] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee8d801f5f354b9da0e0b75d2d4eb108; no voters:
I20260808 12:56:26.444661 25353 leader_election.cc:288] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:56:26.445225 25359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:26.447129 25359 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [term 1 LEADER]: Becoming Leader. State: Replica: ee8d801f5f354b9da0e0b75d2d4eb108, State: Running, Role: LEADER
I20260808 12:56:26.448060 25353 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:56:26.447973 25359 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8d801f5f354b9da0e0b75d2d4eb108" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46647 } }
I20260808 12:56:26.455858 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:26.409511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.24.167.62:46647)
0808 12:56:26.410774 (+ 1263us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:26.410790 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:26.412437 (+ 1647us) server_negotiation.cc:407] Connection header received
0808 12:56:26.423461 (+ 11024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:26.423497 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:26.423977 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:26.424311 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:26.428326 (+ 4015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:26.430747 (+ 2421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:26.433161 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:26.434602 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:26.434877 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:26.443093 (+ 8216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:26.443103 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:26.443119 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:26.443168 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:26.449155 (+ 5987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:26.450487 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:26.450496 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:26.450502 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:26.450700 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:26.451295 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:26.451301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:26.451307 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:26.451968 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:26.452376 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:26.453352 (+ 976us) server_negotiation.cc:299] Negotiation successful
0808 12:56:26.453836 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983,"thread_start_us":644,"threads_started":1}
I20260808 12:56:26.458230 25361 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee8d801f5f354b9da0e0b75d2d4eb108. Latest consensus state: current_term: 1 leader_uuid: "ee8d801f5f354b9da0e0b75d2d4eb108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8d801f5f354b9da0e0b75d2d4eb108" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46647 } } }
I20260808 12:56:26.458729 25361 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [sys.catalog]: This master's current role is: LEADER
I20260808 12:56:26.460150 25360 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee8d801f5f354b9da0e0b75d2d4eb108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8d801f5f354b9da0e0b75d2d4eb108" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46647 } } }
I20260808 12:56:26.460758 25360 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108 [sys.catalog]: This master's current role is: LEADER
I20260808 12:56:26.463096 25369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:56:26.470589 25369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:56:26.481328 25369 catalog_manager.cc:1347] Generated new cluster ID: a09e8c61368e44778560ec0c68abcf1f
I20260808 12:56:26.481448 25369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:56:26.510208 25369 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:56:26.512511 25369 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:56:26.527688 25369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee8d801f5f354b9da0e0b75d2d4eb108: Generated new TSK 0
I20260808 12:56:26.529033 25369 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 12:56:26.580595 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:46647
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:27.106215 25387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:27.106544 25387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:27.106729 25387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:27.114954 25387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:27.115149 25387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:27.115340 25387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:56:27.126608 25387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:46647
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:27.128103 25387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:27.129546 25387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:27.145171 25393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:27.145668 25394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:27.145754 25396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:27.147686 25387 server_base.cc:1034] running on GCE node
I20260808 12:56:27.148530 25387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:27.149668 25387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:27.150935 25387 hybrid_clock.cc:648] HybridClock initialized: now 1786193787150874 us; error 73 us; skew 500 ppm
I20260808 12:56:27.151396 25387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:27 dist-test-slave-tk9z krb5kdc[25252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193787, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:27.155697 25387 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 12:56:27.157379 25387 webserver.cc:460] Webserver started at http://127.24.167.1:42055/ using document root <none> and password file <none>
I20260808 12:56:27.158234 25387 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:27.158515 25387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:27.159022 25387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:27.164211 25387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "c007adcc45b842beb6d9fb09aa4855de"
format_stamp: "Formatted at 2026-08-08 12:56:27 on dist-test-slave-tk9z"
server_key: "8788abaa94d26aeab5a5d0f55bd3672d"
server_key_iv: "f60732f2ac9c577001b3609df60bf08a"
server_key_version: "encryptionkey@0"
I20260808 12:56:27.165221 25387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "c007adcc45b842beb6d9fb09aa4855de"
format_stamp: "Formatted at 2026-08-08 12:56:27 on dist-test-slave-tk9z"
server_key: "8788abaa94d26aeab5a5d0f55bd3672d"
server_key_iv: "f60732f2ac9c577001b3609df60bf08a"
server_key_version: "encryptionkey@0"
I20260808 12:56:27.171938 25387 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260808 12:56:27.176599 25408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:27.178107 25387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 12:56:27.178293 25387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "c007adcc45b842beb6d9fb09aa4855de"
format_stamp: "Formatted at 2026-08-08 12:56:27 on dist-test-slave-tk9z"
server_key: "8788abaa94d26aeab5a5d0f55bd3672d"
server_key_iv: "f60732f2ac9c577001b3609df60bf08a"
server_key_version: "encryptionkey@0"
I20260808 12:56:27.178580 25387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:27.195186 25387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:27.200907 25387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:27.201336 25387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:27.202625 25387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:56:27.204546 25387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:27.204628 25387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:27.204792 25387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:27.204849 25387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:27.265955 25387 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:36317
I20260808 12:56:27.266073 25565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:36317 every 8 connection(s)
I20260808 12:56:27.267467 25387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:56:27.268148 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 25387
I20260808 12:56:27.268401 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:56:27.269519 25244 external_mini_cluster.cc:1428] Setting key ada28180bef840c09f8ffadf71f94d07
Aug 08 12:56:27 dist-test-slave-tk9z krb5kdc[25252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193787, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
I20260808 12:56:27.295126 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:27.271029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:58685 (local address 127.24.167.62:46647)
0808 12:56:27.271841 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:27.271850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:27.273693 (+ 1843us) server_negotiation.cc:407] Connection header received
0808 12:56:27.276465 (+ 2772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:27.276471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:27.276583 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:27.276717 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:27.278786 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:27.279796 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:27.283011 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:27.283777 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:27.283885 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:27.288739 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:27.288743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:27.288747 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:27.288781 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:27.291722 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:27.292660 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:27.292665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:27.292667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:27.292805 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:27.293267 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:27.293271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:27.293273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:27.293549 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:27.293706 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:27.294500 (+ 794us) server_negotiation.cc:299] Negotiation successful
0808 12:56:27.294736 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":418,"threads_started":1}
I20260808 12:56:27.296207 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:27.271903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:46647 (local address 127.24.167.1:58685)
0808 12:56:27.273513 (+ 1610us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:27.273572 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:27.276161 (+ 2589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:27.277006 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:27.277016 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:27.277608 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:27.278559 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:27.278590 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:27.279979 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:27.279988 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:27.282750 (+ 2762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:27.282772 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:27.284082 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:27.284096 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:27.284493 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:27.285294 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:27.285342 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:27.288499 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:27.291948 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:27.291956 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:27.291969 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:27.292526 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:27.292924 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:27.292930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:27.292934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:27.293138 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:27.293680 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:27.293688 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:27.294050 (+ 362us) client_negotiation.cc:769] Sending connection context
0808 12:56:27.294321 (+ 271us) client_negotiation.cc:240] Negotiation successful
0808 12:56:27.294905 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"thread_start_us":951,"threads_started":1}
I20260808 12:56:27.298092 25567 heartbeater.cc:347] Connected to a master server at 127.24.167.62:46647
I20260808 12:56:27.298552 25567 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:27.299429 25567 heartbeater.cc:511] Master 127.24.167.62:46647 requested a full tablet report, sending...
I20260808 12:56:27.302073 25302 ts_manager.cc:194] Registered new tserver with Master: c007adcc45b842beb6d9fb09aa4855de (127.24.167.1:36317)
I20260808 12:56:27.304049 25302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:58685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 12:56:27.315985 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:46647
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:27.767715 25574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:27.768110 25574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:27.768319 25574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:27.777235 25574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:27.777400 25574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:27.777637 25574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:56:27.789175 25574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:46647
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:27.790798 25574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:27.792439 25574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:27.810501 25587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:27.810827 25584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:27.811054 25582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:27.813387 25574 server_base.cc:1034] running on GCE node
I20260808 12:56:27.814242 25574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:27.815462 25574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:27.816685 25574 hybrid_clock.cc:648] HybridClock initialized: now 1786193787816641 us; error 57 us; skew 500 ppm
I20260808 12:56:27.817085 25574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:27 dist-test-slave-tk9z krb5kdc[25252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193787, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:27.820917 25574 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 12:56:27.822754 25574 webserver.cc:460] Webserver started at http://127.24.167.2:45087/ using document root <none> and password file <none>
I20260808 12:56:27.823597 25574 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:27.823740 25574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:27.824126 25574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:27.827680 25574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "77b32d9113504ff09f01327592d4d899"
format_stamp: "Formatted at 2026-08-08 12:56:27 on dist-test-slave-tk9z"
server_key: "de5501a2be5692261625655b7a03642d"
server_key_iv: "a0eb3372da4eb52a7eeffec8257b745d"
server_key_version: "encryptionkey@0"
I20260808 12:56:27.828516 25574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "77b32d9113504ff09f01327592d4d899"
format_stamp: "Formatted at 2026-08-08 12:56:27 on dist-test-slave-tk9z"
server_key: "de5501a2be5692261625655b7a03642d"
server_key_iv: "a0eb3372da4eb52a7eeffec8257b745d"
server_key_version: "encryptionkey@0"
I20260808 12:56:27.834642 25574 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260808 12:56:27.839354 25595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:27.841037 25574 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 12:56:27.841260 25574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "77b32d9113504ff09f01327592d4d899"
format_stamp: "Formatted at 2026-08-08 12:56:27 on dist-test-slave-tk9z"
server_key: "de5501a2be5692261625655b7a03642d"
server_key_iv: "a0eb3372da4eb52a7eeffec8257b745d"
server_key_version: "encryptionkey@0"
I20260808 12:56:27.841607 25574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:27.870720 25574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:27.875667 25574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:27.876112 25574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:27.877359 25574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:56:27.878999 25574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:27.879077 25574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:27.879212 25574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:27.879266 25574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:27.948096 25574 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:43399
I20260808 12:56:27.948479 25750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:43399 every 8 connection(s)
I20260808 12:56:27.949738 25574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:56:27.957716 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 25574
I20260808 12:56:27.958056 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:56:27.958693 25244 external_mini_cluster.cc:1428] Setting key f47f2b88947cb80c3c0f4f7150294e07
Aug 08 12:56:27 dist-test-slave-tk9z krb5kdc[25252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193787, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
I20260808 12:56:27.981609 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:27.953577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:60607 (local address 127.24.167.62:46647)
0808 12:56:27.954358 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:27.954363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:27.956084 (+ 1721us) server_negotiation.cc:407] Connection header received
0808 12:56:27.959181 (+ 3097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:27.959188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:27.959379 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:27.959528 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:27.962227 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:27.963460 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:27.966335 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:27.967692 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:27.967853 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:27.972251 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:27.972263 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:27.972271 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:27.972326 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:27.977420 (+ 5094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:27.978352 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:27.978359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:27.978363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:27.978595 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:27.979182 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:27.979189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:27.979195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:27.979696 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:27.979895 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:27.980782 (+ 887us) server_negotiation.cc:299] Negotiation successful
0808 12:56:27.981066 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":529,"threads_started":1}
I20260808 12:56:27.982613 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:27.954561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:46647 (local address 127.24.167.2:60607)
0808 12:56:27.955890 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:27.955942 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:27.958853 (+ 2911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:27.959942 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:27.959956 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:27.960813 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:27.962019 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:27.962055 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:27.963630 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:27.963640 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:27.966178 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:27.966196 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:27.967993 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:27.968000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:27.968379 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:27.969095 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:27.969117 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:27.971984 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:27.977581 (+ 5597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:27.977589 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:27.977603 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:27.978203 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:27.978740 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:27.978743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:27.978746 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:27.979047 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:27.979917 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:27.979924 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:27.980308 (+ 384us) client_negotiation.cc:769] Sending connection context
0808 12:56:27.980589 (+ 281us) client_negotiation.cc:240] Negotiation successful
0808 12:56:27.981087 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":623,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
I20260808 12:56:27.984433 25751 heartbeater.cc:347] Connected to a master server at 127.24.167.62:46647
I20260808 12:56:27.984807 25751 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:27.985618 25751 heartbeater.cc:511] Master 127.24.167.62:46647 requested a full tablet report, sending...
I20260808 12:56:27.987525 25302 ts_manager.cc:194] Registered new tserver with Master: 77b32d9113504ff09f01327592d4d899 (127.24.167.2:43399)
I20260808 12:56:27.989086 25302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:60607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 12:56:28.009240 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:46647
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 12:56:28.307490 25567 heartbeater.cc:503] Master 127.24.167.62:46647 was elected leader, sending a full tablet report...
W20260808 12:56:28.544499 25759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:28.544876 25759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:28.545107 25759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:28.555372 25759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:28.555572 25759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:28.555778 25759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:56:28.568486 25759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:46647
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:28.570032 25759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:28.571728 25759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:28.589754 25766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:28.590102 25767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:28.590279 25769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:28.591995 25759 server_base.cc:1034] running on GCE node
I20260808 12:56:28.592723 25759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:28.593915 25759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:28.595198 25759 hybrid_clock.cc:648] HybridClock initialized: now 1786193788595103 us; error 53 us; skew 500 ppm
I20260808 12:56:28.595556 25759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:28 dist-test-slave-tk9z krb5kdc[25252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193788, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:28.600414 25759 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 12:56:28.602486 25759 webserver.cc:460] Webserver started at http://127.24.167.3:43519/ using document root <none> and password file <none>
I20260808 12:56:28.603353 25759 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:28.603546 25759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:28.603982 25759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:28.607717 25759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "a1f413e0e5d942a38c5247e5ac51a399"
format_stamp: "Formatted at 2026-08-08 12:56:28 on dist-test-slave-tk9z"
server_key: "d3bebf7c54e3ef20b053ee65ddb6530c"
server_key_iv: "745af7985c0cdcb7bb14285b5d1bb9fa"
server_key_version: "encryptionkey@0"
I20260808 12:56:28.608640 25759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "a1f413e0e5d942a38c5247e5ac51a399"
format_stamp: "Formatted at 2026-08-08 12:56:28 on dist-test-slave-tk9z"
server_key: "d3bebf7c54e3ef20b053ee65ddb6530c"
server_key_iv: "745af7985c0cdcb7bb14285b5d1bb9fa"
server_key_version: "encryptionkey@0"
I20260808 12:56:28.615146 25759 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260808 12:56:28.619503 25778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:28.620909 25759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260808 12:56:28.621140 25759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "a1f413e0e5d942a38c5247e5ac51a399"
format_stamp: "Formatted at 2026-08-08 12:56:28 on dist-test-slave-tk9z"
server_key: "d3bebf7c54e3ef20b053ee65ddb6530c"
server_key_iv: "745af7985c0cdcb7bb14285b5d1bb9fa"
server_key_version: "encryptionkey@0"
I20260808 12:56:28.621415 25759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:28.639246 25759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:28.645139 25759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:28.645608 25759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:28.647439 25759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:56:28.649744 25759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:28.649971 25759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:28.650211 25759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:28.650334 25759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:28.719475 25759 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:43169
I20260808 12:56:28.719489 25924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:43169 every 8 connection(s)
I20260808 12:56:28.721211 25759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:56:28.724378 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 25759
I20260808 12:56:28.724663 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:56:28.725453 25244 external_mini_cluster.cc:1428] Setting key f99495567ec9c50a9a79c44ff79c7926
Aug 08 12:56:28 dist-test-slave-tk9z krb5kdc[25252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193788, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:28.748051 25930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:28.725052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:42653 (local address 127.24.167.62:46647)
0808 12:56:28.725814 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:28.725819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:28.727737 (+ 1918us) server_negotiation.cc:407] Connection header received
0808 12:56:28.730819 (+ 3082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:28.730823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:28.730983 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:28.731087 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:28.732930 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:28.733861 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:28.735707 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:28.736276 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:28.736379 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:28.740703 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:28.740708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:28.740711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:28.740746 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:28.744816 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:28.745600 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:28.745604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:28.745606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:28.745722 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:28.746239 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:28.746243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:28.746245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:28.746580 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:28.746704 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:28.747528 (+ 824us) server_negotiation.cc:299] Negotiation successful
0808 12:56:28.747702 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":485,"threads_started":1}
I20260808 12:56:28.749183 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:28.725984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:46647 (local address 127.24.167.3:42653)
0808 12:56:28.727523 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:28.727612 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:28.730574 (+ 2962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:28.731304 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:28.731315 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:28.731980 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:28.732731 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:28.732770 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:28.733996 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:28.734000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:28.735567 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:28.735580 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:28.736487 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:28.736491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:28.736750 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:28.737616 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:28.737640 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:28.740411 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:28.744943 (+ 4532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:28.744951 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:28.744964 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:28.745474 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:28.745828 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:28.745831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:28.745834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:28.746131 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:28.746690 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:28.746695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:28.747090 (+ 395us) client_negotiation.cc:769] Sending connection context
0808 12:56:28.747386 (+ 296us) client_negotiation.cc:240] Negotiation successful
0808 12:56:28.747855 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"thread_start_us":805,"threads_started":1}
I20260808 12:56:28.750672 25925 heartbeater.cc:347] Connected to a master server at 127.24.167.62:46647
I20260808 12:56:28.751039 25925 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:28.751981 25925 heartbeater.cc:511] Master 127.24.167.62:46647 requested a full tablet report, sending...
I20260808 12:56:28.753543 25302 ts_manager.cc:194] Registered new tserver with Master: a1f413e0e5d942a38c5247e5ac51a399 (127.24.167.3:43169)
I20260808 12:56:28.754683 25302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:42653
I20260808 12:56:28.755716 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 12:56:28 dist-test-slave-tk9z krb5kdc[25252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193788, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 12:56:28 dist-test-slave-tk9z krb5kdc[25252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193788, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:56:28.809003 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:28.790109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.24.167.1:36317)
0808 12:56:28.791003 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:28.791012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:28.791131 (+ 119us) server_negotiation.cc:407] Connection header received
0808 12:56:28.791286 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:28.791293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:28.791540 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:28.791683 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:28.794024 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:28.795130 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:28.796727 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:28.797298 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:28.797407 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:28.802077 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:28.802089 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:28.802104 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:28.802140 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:28.805950 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:28.806738 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:28.806745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:28.806747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:28.806920 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:28.807265 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:28.807268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:28.807270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:28.807721 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:28.808015 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:28.808273 (+ 258us) server_negotiation.cc:299] Negotiation successful
0808 12:56:28.808506 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":524,"threads_started":1}
W20260808 12:56:28.810501 25449 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50874
Aug 08 12:56:28 dist-test-slave-tk9z krb5kdc[25252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 12:56:28 dist-test-slave-tk9z krb5kdc[25252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:56:28.860836 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:28.844622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.24.167.1:36317)
0808 12:56:28.844895 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:28.844903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:28.845288 (+ 385us) server_negotiation.cc:407] Connection header received
0808 12:56:28.845522 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:28.845526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:28.845595 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:28.845705 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:28.847450 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:28.848265 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:28.849904 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:28.850338 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:28.850519 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:28.855081 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:28.855086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:28.855090 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:28.855129 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:28.858166 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:28.858879 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:28.858882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:28.858885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:28.859103 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:28.859422 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:28.859425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:28.859428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:28.859717 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:28.859962 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:28.860186 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 12:56:28.860410 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 12:56:28.862757 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 25387
I20260808 12:56:28.878556 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:28.864285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.24.167.1:36317)
0808 12:56:28.864476 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:28.864480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:28.864497 (+ 17us) server_negotiation.cc:407] Connection header received
0808 12:56:28.864684 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:28.864688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:28.864750 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:28.864862 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:28.866033 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:28.866989 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:28.868263 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:28.868827 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:28.868927 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:28.871972 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:28.871978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:28.871981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:28.872012 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:28.875896 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:28.876669 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:28.876673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:28.876675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:28.876876 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:28.877251 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:28.877254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:28.877256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:28.877508 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:28.877690 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:28.877928 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 12:56:28.878101 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 12:56:28.879596 25556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:56:28.981027 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 25387
I20260808 12:56:28.992574 25751 heartbeater.cc:503] Master 127.24.167.62:46647 was elected leader, sending a full tablet report...
I20260808 12:56:29.009575 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 25574
Aug 08 12:56:29 dist-test-slave-tk9z krb5kdc[25252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 12:56:29.032071 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:29.011089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.24.167.2:43399)
0808 12:56:29.011918 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:29.011929 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:29.011978 (+ 49us) server_negotiation.cc:407] Connection header received
0808 12:56:29.012078 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:29.012090 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:29.012406 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:29.012613 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:29.014286 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:29.016222 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:29.018249 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:29.019369 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:29.019533 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:29.024859 (+ 5326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:29.024880 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:29.024904 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:29.024975 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:29.029188 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:29.029858 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:29.029865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:29.029867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:29.030015 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:29.030502 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:29.030505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:29.030507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:29.030861 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:29.031100 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:29.031423 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 12:56:29.031639 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":427,"threads_started":1}
I20260808 12:56:29.033519 25745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:56:29.118296 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 25574
I20260808 12:56:29.142917 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 25759
Aug 08 12:56:29 dist-test-slave-tk9z krb5kdc[25252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 12:56:29.160158 25950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:29.143998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58144 (local address 127.24.167.3:43169)
0808 12:56:29.144604 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:29.144611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:29.144647 (+ 36us) server_negotiation.cc:407] Connection header received
0808 12:56:29.144718 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:29.144726 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:29.145001 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:29.145156 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:29.146494 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:29.147661 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:29.148976 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:29.149686 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:29.149873 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:29.154246 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:29.154260 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:29.154278 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:29.154325 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:29.157333 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:29.158034 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:29.158042 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:29.158044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:29.158192 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:29.158598 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:29.158601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:29.158603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:29.158993 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:29.159223 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:29.159497 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 12:56:29.159708 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":262,"threads_started":1}
I20260808 12:56:29.161463 25919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:56:29.247550 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 25759
I20260808 12:56:29.270478 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 25268
I20260808 12:56:29.271744 25346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:56:29.376544 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 25268
2026-08-08T12:56:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5276 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:56:29 dist-test-slave-tk9z krb5kdc[25957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:56:29 dist-test-slave-tk9z krb5kdc[25957](info): set up 2 sockets
Aug 08 12:56:29 dist-test-slave-tk9z krb5kdc[25957](info): commencing operation
krb5kdc: starting...
W20260808 12:56:30.661597 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:56:30 dist-test-slave-tk9z krb5kdc[25957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:56:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:56:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 12:56:30.854609 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:42525
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:42525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:31.195779 25975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:31.196058 25975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:31.196226 25975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:31.203357 25975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:56:31.203461 25975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:31.203558 25975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:31.203655 25975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:56:31.203730 25975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:56:31.213689 25975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:42525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:42525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:31.215096 25975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:31.216481 25975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:31.228286 25981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:31.228330 25982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:31.228385 25984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:31.229550 25975 server_base.cc:1034] running on GCE node
I20260808 12:56:31.230468 25975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:31.231884 25975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:31.233143 25975 hybrid_clock.cc:648] HybridClock initialized: now 1786193791233104 us; error 53 us; skew 500 ppm
I20260808 12:56:31.233525 25975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:31 dist-test-slave-tk9z krb5kdc[25957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193791, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:31.237071 25975 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:56:31.238727 25975 webserver.cc:460] Webserver started at http://127.24.167.62:41413/ using document root <none> and password file <none>
I20260808 12:56:31.239447 25975 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:31.239602 25975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:31.239935 25975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:31.243088 25975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "51519718bd314772b930ed2951bc27e5"
format_stamp: "Formatted at 2026-08-08 12:56:31 on dist-test-slave-tk9z"
server_key: "c31b68fa6d6c557de09de0d71fb3aa74"
server_key_iv: "032dd4c1e6a518d0cd19e04361f389a1"
server_key_version: "encryptionkey@0"
I20260808 12:56:31.243844 25975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "51519718bd314772b930ed2951bc27e5"
format_stamp: "Formatted at 2026-08-08 12:56:31 on dist-test-slave-tk9z"
server_key: "c31b68fa6d6c557de09de0d71fb3aa74"
server_key_iv: "032dd4c1e6a518d0cd19e04361f389a1"
server_key_version: "encryptionkey@0"
I20260808 12:56:31.248988 25975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 12:56:31.252348 25994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:31.253599 25975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 12:56:31.253798 25975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "51519718bd314772b930ed2951bc27e5"
format_stamp: "Formatted at 2026-08-08 12:56:31 on dist-test-slave-tk9z"
server_key: "c31b68fa6d6c557de09de0d71fb3aa74"
server_key_iv: "032dd4c1e6a518d0cd19e04361f389a1"
server_key_version: "encryptionkey@0"
I20260808 12:56:31.254005 25975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:31.282784 25975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:31.286755 25975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:31.287062 25975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:31.306258 25975 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:42525
I20260808 12:56:31.306282 26066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:42525 every 8 connection(s)
I20260808 12:56:31.308197 25975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:56:31.313083 26068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:31.313912 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 25975
I20260808 12:56:31.314165 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:56:31.315163 25244 external_mini_cluster.cc:1428] Setting key e93142d047467f57cab7cafd3599805e
I20260808 12:56:31.328087 26068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5: Bootstrap starting.
I20260808 12:56:31.331367 26068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5: Neither blocks nor log segments found. Creating new log.
Aug 08 12:56:31 dist-test-slave-tk9z krb5kdc[25957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:31.332435 26068 log.cc:824] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5: Log is configured to *not* fsync() on all Append() calls
I20260808 12:56:31.335335 26068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5: No bootstrap required, opened a new log
I20260808 12:56:31.340507 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:31.318620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.24.167.62:42525)
0808 12:56:31.319510 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:31.319532 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:31.319594 (+ 62us) server_negotiation.cc:407] Connection header received
0808 12:56:31.321827 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:31.321851 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:31.322300 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:31.322812 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:31.324843 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:31.326647 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:31.327917 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:31.328679 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:31.328887 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:31.333041 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:31.333049 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:31.333064 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:31.333099 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:31.336296 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:31.336976 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:31.336988 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:31.336997 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:31.337172 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:31.337562 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:31.337569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:31.337577 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:31.338142 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:31.338545 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:31.339034 (+ 489us) server_negotiation.cc:299] Negotiation successful
0808 12:56:31.339490 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":370,"threads_started":1}
I20260808 12:56:31.343282 26068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51519718bd314772b930ed2951bc27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42525 } }
I20260808 12:56:31.343755 26068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:31.343863 26068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51519718bd314772b930ed2951bc27e5, State: Initialized, Role: FOLLOWER
I20260808 12:56:31.344395 26068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51519718bd314772b930ed2951bc27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42525 } }
I20260808 12:56:31.344647 26068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:56:31.344806 26068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:56:31.344970 26068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:31.347714 26068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51519718bd314772b930ed2951bc27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42525 } }
I20260808 12:56:31.348204 26068 leader_election.cc:302] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51519718bd314772b930ed2951bc27e5; no voters:
I20260808 12:56:31.348768 26068 leader_election.cc:288] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:56:31.348896 26076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:31.349647 26076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [term 1 LEADER]: Becoming Leader. State: Replica: 51519718bd314772b930ed2951bc27e5, State: Running, Role: LEADER
I20260808 12:56:31.350208 26076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51519718bd314772b930ed2951bc27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42525 } }
I20260808 12:56:31.350934 26068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:56:31.353163 26079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51519718bd314772b930ed2951bc27e5. Latest consensus state: current_term: 1 leader_uuid: "51519718bd314772b930ed2951bc27e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51519718bd314772b930ed2951bc27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42525 } } }
I20260808 12:56:31.353241 26078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51519718bd314772b930ed2951bc27e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51519718bd314772b930ed2951bc27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42525 } } }
I20260808 12:56:31.353511 26079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [sys.catalog]: This master's current role is: LEADER
I20260808 12:56:31.353561 26078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5 [sys.catalog]: This master's current role is: LEADER
I20260808 12:56:31.355190 26085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:56:31.361963 26085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:56:31.375344 26085 catalog_manager.cc:1347] Generated new cluster ID: 75196d534ed94f9c962bb30e9f71e028
I20260808 12:56:31.375463 26085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:56:31.398224 26085 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:56:31.399677 26085 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:56:31.409534 26085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51519718bd314772b930ed2951bc27e5: Generated new TSK 0
I20260808 12:56:31.410490 26085 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 12:56:31.464390 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42525
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:31.801371 26107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:31.801757 26107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:31.801942 26107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:31.809556 26107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:31.809731 26107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:31.809921 26107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:56:31.820004 26107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42525
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:31.821391 26107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:31.822815 26107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:31.837200 26114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:31.837246 26113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:31.837167 26117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:31.838006 26107 server_base.cc:1034] running on GCE node
I20260808 12:56:31.838763 26107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:31.839818 26107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:31.841051 26107 hybrid_clock.cc:648] HybridClock initialized: now 1786193791840986 us; error 55 us; skew 500 ppm
I20260808 12:56:31.841377 26107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:31 dist-test-slave-tk9z krb5kdc[25957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193791, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:31.844663 26107 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 12:56:31.846057 26107 webserver.cc:460] Webserver started at http://127.24.167.1:36567/ using document root <none> and password file <none>
I20260808 12:56:31.846827 26107 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:31.846972 26107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:31.847296 26107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:31.850175 26107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "263043659f9640c9a47733fca2fbe4e8"
format_stamp: "Formatted at 2026-08-08 12:56:31 on dist-test-slave-tk9z"
server_key: "dc0b8c64097cea6958d048375eef8f4b"
server_key_iv: "92389c6bf37f74ac84a26a3a3d6a3dc2"
server_key_version: "encryptionkey@0"
I20260808 12:56:31.851094 26107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "263043659f9640c9a47733fca2fbe4e8"
format_stamp: "Formatted at 2026-08-08 12:56:31 on dist-test-slave-tk9z"
server_key: "dc0b8c64097cea6958d048375eef8f4b"
server_key_iv: "92389c6bf37f74ac84a26a3a3d6a3dc2"
server_key_version: "encryptionkey@0"
I20260808 12:56:31.857170 26107 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260808 12:56:31.860797 26128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:31.862165 26107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 12:56:31.862354 26107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "263043659f9640c9a47733fca2fbe4e8"
format_stamp: "Formatted at 2026-08-08 12:56:31 on dist-test-slave-tk9z"
server_key: "dc0b8c64097cea6958d048375eef8f4b"
server_key_iv: "92389c6bf37f74ac84a26a3a3d6a3dc2"
server_key_version: "encryptionkey@0"
I20260808 12:56:31.862622 26107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:31.900245 26107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:31.904417 26107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:31.904738 26107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:31.905761 26107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:56:31.907238 26107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:31.907357 26107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:31.907510 26107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:31.907583 26107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:31.942713 26107 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:44841
I20260808 12:56:31.942806 26278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:44841 every 8 connection(s)
I20260808 12:56:31.944053 26107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:56:31.945591 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 26107
I20260808 12:56:31.945823 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:56:31.946506 25244 external_mini_cluster.cc:1428] Setting key f621a64e2356c04372fa621d74c5a561
Aug 08 12:56:31 dist-test-slave-tk9z krb5kdc[25957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193791, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:31.965317 26283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:31.947945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:40067 (local address 127.24.167.62:42525)
0808 12:56:31.948886 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:31.948893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:31.948914 (+ 21us) server_negotiation.cc:407] Connection header received
0808 12:56:31.950333 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:31.950338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:31.950458 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:31.950575 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:31.952271 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:31.953025 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:31.954622 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:31.955020 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:31.955158 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:31.959191 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:31.959196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:31.959200 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:31.959234 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:31.961833 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:31.962873 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:31.962877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:31.962879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:31.963039 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:31.963545 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:31.963548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:31.963550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:31.963832 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:31.963940 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:31.964899 (+ 959us) server_negotiation.cc:299] Negotiation successful
0808 12:56:31.965068 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":710,"threads_started":1}
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
I20260808 12:56:31.966849 26282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:31.947379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42525 (local address 127.24.167.1:40067)
0808 12:56:31.948197 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:31.948251 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:31.950043 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:31.950812 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:31.950820 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:31.951462 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:31.952092 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:31.952119 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:31.953191 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:31.953201 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:31.954484 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:31.954497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:31.955255 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:31.955265 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:31.955506 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:31.956225 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:31.956247 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:31.958950 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:31.961992 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:31.962005 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:31.962025 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:31.962754 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:31.963155 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:31.963162 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:31.963166 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:31.963419 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:31.963959 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:31.963969 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:31.964467 (+ 498us) client_negotiation.cc:769] Sending connection context
0808 12:56:31.964791 (+ 324us) client_negotiation.cc:240] Negotiation successful
0808 12:56:31.965410 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":327,"threads_started":1}
I20260808 12:56:31.968418 26280 heartbeater.cc:347] Connected to a master server at 127.24.167.62:42525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
I20260808 12:56:31.968766 26280 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:31.969532 26280 heartbeater.cc:511] Master 127.24.167.62:42525 requested a full tablet report, sending...
I20260808 12:56:31.972052 26022 ts_manager.cc:194] Registered new tserver with Master: 263043659f9640c9a47733fca2fbe4e8 (127.24.167.1:44841)
I20260808 12:56:31.974016 26022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:40067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 12:56:31.992357 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42525
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:32.333884 26286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:32.334157 26286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:32.334255 26286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:32.341562 26286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:32.341675 26286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:32.341790 26286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:56:32.351775 26286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42525
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:32.353077 26286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:32.354544 26286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:32.368142 26293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:32.368239 26292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:32.368222 26297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:32.369191 26286 server_base.cc:1034] running on GCE node
I20260808 12:56:32.369791 26286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:32.370829 26286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:32.372053 26286 hybrid_clock.cc:648] HybridClock initialized: now 1786193792372000 us; error 43 us; skew 500 ppm
I20260808 12:56:32.372385 26286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:32 dist-test-slave-tk9z krb5kdc[25957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193792, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:32.375671 26286 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 12:56:32.377163 26286 webserver.cc:460] Webserver started at http://127.24.167.2:34591/ using document root <none> and password file <none>
I20260808 12:56:32.377848 26286 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:32.378003 26286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:32.378330 26286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:32.381258 26286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "a2dd446b021447389fbf062471c09101"
format_stamp: "Formatted at 2026-08-08 12:56:32 on dist-test-slave-tk9z"
server_key: "19016f427937cc52ea61f2eb2109a857"
server_key_iv: "634c1ad6ebf1585b1e38e2ddb141cd4f"
server_key_version: "encryptionkey@0"
I20260808 12:56:32.382145 26286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "a2dd446b021447389fbf062471c09101"
format_stamp: "Formatted at 2026-08-08 12:56:32 on dist-test-slave-tk9z"
server_key: "19016f427937cc52ea61f2eb2109a857"
server_key_iv: "634c1ad6ebf1585b1e38e2ddb141cd4f"
server_key_version: "encryptionkey@0"
I20260808 12:56:32.387488 26286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 12:56:32.391134 26306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:32.392604 26286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 12:56:32.392791 26286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "a2dd446b021447389fbf062471c09101"
format_stamp: "Formatted at 2026-08-08 12:56:32 on dist-test-slave-tk9z"
server_key: "19016f427937cc52ea61f2eb2109a857"
server_key_iv: "634c1ad6ebf1585b1e38e2ddb141cd4f"
server_key_version: "encryptionkey@0"
I20260808 12:56:32.393009 26286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:32.412793 26286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:32.417373 26286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:32.417696 26286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:32.418759 26286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:56:32.420154 26286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:32.420261 26286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:32.420413 26286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:32.420495 26286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:32.454982 26286 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:35917
I20260808 12:56:32.455047 26458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:35917 every 8 connection(s)
I20260808 12:56:32.456321 26286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:56:32.463282 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 26286
I20260808 12:56:32.463500 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:56:32.464080 25244 external_mini_cluster.cc:1428] Setting key 332b4568531de678c04bd8c10b23827d
Aug 08 12:56:32 dist-test-slave-tk9z krb5kdc[25957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193792, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:32.480311 26283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:32.459039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:38285 (local address 127.24.167.62:42525)
0808 12:56:32.459225 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:32.459230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:32.460469 (+ 1239us) server_negotiation.cc:407] Connection header received
0808 12:56:32.462220 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:32.462224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:32.462295 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:32.462457 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:32.464568 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:32.465382 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:32.468270 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:32.468717 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:32.468818 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:32.473487 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:32.473495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:32.473503 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:32.473554 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:32.476362 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:32.477383 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:32.477386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:32.477389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:32.477544 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:32.478158 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:32.478163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:32.478165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:32.478585 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:32.478729 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:32.479742 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0808 12:56:32.480026 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 12:56:32.481874 26461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:32.459575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42525 (local address 127.24.167.2:38285)
0808 12:56:32.460296 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:32.460340 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:32.461962 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:32.462662 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:32.462670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:32.463549 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:32.464372 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:32.464399 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:32.465567 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:32.465571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:32.468078 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:32.468108 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:32.468974 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:32.468984 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:32.469352 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:32.470181 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:32.470224 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:32.473195 (+ 2971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:32.476521 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:32.476535 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:32.476556 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:32.477253 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:32.477664 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:32.477672 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:32.477678 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:32.477990 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:32.478735 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:32.478746 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:32.479264 (+ 518us) client_negotiation.cc:769] Sending connection context
0808 12:56:32.479614 (+ 350us) client_negotiation.cc:240] Negotiation successful
0808 12:56:32.480264 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":280,"threads_started":1}
I20260808 12:56:32.483498 26459 heartbeater.cc:347] Connected to a master server at 127.24.167.62:42525
I20260808 12:56:32.483830 26459 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
I20260808 12:56:32.484674 26459 heartbeater.cc:511] Master 127.24.167.62:42525 requested a full tablet report, sending...
I20260808 12:56:32.486335 26022 ts_manager.cc:194] Registered new tserver with Master: a2dd446b021447389fbf062471c09101 (127.24.167.2:35917)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
I20260808 12:56:32.487438 26022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:38285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 12:56:32.513473 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42525
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:32.851444 26466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:32.851732 26466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:32.851922 26466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:32.859333 26466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:32.859448 26466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:32.859623 26466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:56:32.869666 26466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42525
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:32.871057 26466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:32.872411 26466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:32.886010 26474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:32.885885 26479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:32.885918 26475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:32.886854 26466 server_base.cc:1034] running on GCE node
I20260808 12:56:32.887462 26466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:32.888528 26466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:32.889735 26466 hybrid_clock.cc:648] HybridClock initialized: now 1786193792889686 us; error 37 us; skew 500 ppm
I20260808 12:56:32.890125 26466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:32 dist-test-slave-tk9z krb5kdc[25957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193792, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:32.893813 26466 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 12:56:32.895517 26466 webserver.cc:460] Webserver started at http://127.24.167.3:45081/ using document root <none> and password file <none>
I20260808 12:56:32.896255 26466 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:32.896356 26466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:32.896731 26466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:32.899914 26466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "9b30f33bff564a71a3be525b71df12bf"
format_stamp: "Formatted at 2026-08-08 12:56:32 on dist-test-slave-tk9z"
server_key: "9b8282fcbbc08023ab1a561b1c9c509b"
server_key_iv: "d2e93eb6c32007f03c8e5b258b4003aa"
server_key_version: "encryptionkey@0"
I20260808 12:56:32.900748 26466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "9b30f33bff564a71a3be525b71df12bf"
format_stamp: "Formatted at 2026-08-08 12:56:32 on dist-test-slave-tk9z"
server_key: "9b8282fcbbc08023ab1a561b1c9c509b"
server_key_iv: "d2e93eb6c32007f03c8e5b258b4003aa"
server_key_version: "encryptionkey@0"
I20260808 12:56:32.906239 26466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 12:56:32.911509 26488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:32.913301 26466 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260808 12:56:32.913483 26466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "9b30f33bff564a71a3be525b71df12bf"
format_stamp: "Formatted at 2026-08-08 12:56:32 on dist-test-slave-tk9z"
server_key: "9b8282fcbbc08023ab1a561b1c9c509b"
server_key_iv: "d2e93eb6c32007f03c8e5b258b4003aa"
server_key_version: "encryptionkey@0"
I20260808 12:56:32.913709 26466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:32.953606 26466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:32.958143 26466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:32.958514 26466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:32.959630 26466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:56:32.961037 26466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:32.961148 26466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:32.961302 26466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:32.961375 26466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:32.977191 26280 heartbeater.cc:503] Master 127.24.167.62:42525 was elected leader, sending a full tablet report...
I20260808 12:56:33.003979 26466 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:43407
I20260808 12:56:33.004232 26637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:43407 every 8 connection(s)
I20260808 12:56:33.005671 26466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:56:33.014592 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 26466
I20260808 12:56:33.014860 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:56:33.015383 25244 external_mini_cluster.cc:1428] Setting key b1a8a8d691eaaa0981307c3136b67ab1
Aug 08 12:56:33 dist-test-slave-tk9z krb5kdc[25957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193792, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:33.034638 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.009151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:34599 (local address 127.24.167.62:42525)
0808 12:56:33.010016 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:33.010021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:33.011349 (+ 1328us) server_negotiation.cc:407] Connection header received
0808 12:56:33.014209 (+ 2860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:33.014216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:33.014341 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:33.014514 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:33.017174 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.018735 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.020958 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.021515 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.021616 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.026149 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:33.026154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:33.026157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:33.026191 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:33.030083 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.031213 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.031218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.031221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.031429 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.032184 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.032193 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.032204 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.032763 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:33.032902 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:33.033801 (+ 899us) server_negotiation.cc:299] Negotiation successful
0808 12:56:33.034026 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":552,"threads_started":1}
I20260808 12:56:33.035706 26643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.010026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42525 (local address 127.24.167.3:34599)
0808 12:56:33.011164 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:33.011212 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:33.013895 (+ 2683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:33.014830 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:33.014839 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:33.015788 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:33.016954 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:33.016986 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.018950 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.018957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:33.020804 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:33.020817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.021753 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.021758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:33.021977 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.022781 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:33.022807 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:33.025932 (+ 3125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:33.030260 (+ 4328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:33.030291 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:33.030344 (+ 53us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:33.031087 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:33.031590 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:33.031594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:33.031597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:33.031957 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:33.032933 (+ 976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:33.032940 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:33.033388 (+ 448us) client_negotiation.cc:769] Sending connection context
0808 12:56:33.033662 (+ 274us) client_negotiation.cc:240] Negotiation successful
0808 12:56:33.034180 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":664,"threads_started":1}
I20260808 12:56:33.037534 26638 heartbeater.cc:347] Connected to a master server at 127.24.167.62:42525
I20260808 12:56:33.037988 26638 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:33.039171 26638 heartbeater.cc:511] Master 127.24.167.62:42525 requested a full tablet report, sending...
I20260808 12:56:33.041141 26022 ts_manager.cc:194] Registered new tserver with Master: 9b30f33bff564a71a3be525b71df12bf (127.24.167.3:43407)
I20260808 12:56:33.042275 26022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:34599
I20260808 12:56:33.045282 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 12:56:33.071098 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.054532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.24.167.62:42525)
0808 12:56:33.054735 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:33.054744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:33.055476 (+ 732us) server_negotiation.cc:407] Connection header received
0808 12:56:33.055734 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:33.055738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:33.055808 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:33.055926 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:33.058091 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.058943 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.060281 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.060881 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.061032 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.064288 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:33.064293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:33.064297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:33.064332 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:33.067878 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.068559 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.068565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.068569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.068788 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.069180 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.069188 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.069191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.069640 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:33.070007 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:33.070345 (+ 338us) server_negotiation.cc:299] Negotiation successful
0808 12:56:33.070655 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 12:56:33.079759 26022 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:56:33.090556 26022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:56:33.130957 26662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.105856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:44841 (local address 127.0.0.1:52202)
0808 12:56:33.116877 (+ 11021us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:33.116937 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:33.117199 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:33.121848 (+ 4649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:33.121856 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:33.121889 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:33.122559 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:33.122571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.125324 (+ 2753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.125330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:33.128030 (+ 2700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:33.128042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.129665 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.129672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:33.129859 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.129922 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 12:56:33.130054 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 12:56:33.130328 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10600,"mutex_wait_us":37,"thread_start_us":14620,"threads_started":1}
I20260808 12:56:33.131084 26660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.106319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.24.167.1:44841)
0808 12:56:33.120783 (+ 14464us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:33.120792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:33.120864 (+ 72us) server_negotiation.cc:407] Connection header received
0808 12:56:33.121031 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:33.121042 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:33.121389 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:33.121706 (+ 317us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:33.122806 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.125124 (+ 2318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.128226 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.129418 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.129584 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.129855 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:33.130211 (+ 356us) server_negotiation.cc:299] Negotiation successful
0808 12:56:33.130482 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14228,"thread_start_us":14602,"threads_started":1}
I20260808 12:56:33.133527 26663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.107742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50786 (local address 127.24.167.3:43407)
0808 12:56:33.118259 (+ 10517us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:33.118272 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:33.122085 (+ 3813us) server_negotiation.cc:407] Connection header received
0808 12:56:33.122193 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:33.122201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:33.122495 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:33.122721 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:33.124321 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.126454 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.129781 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.131896 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.132030 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.132379 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:33.132878 (+ 499us) server_negotiation.cc:299] Negotiation successful
0808 12:56:33.133053 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10278,"thread_start_us":10640,"threads_started":1}
I20260808 12:56:33.133575 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.106700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:43407 (local address 127.0.0.1:50786)
0808 12:56:33.121694 (+ 14994us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:33.121713 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:33.121894 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:33.122725 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:33.122729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:33.122757 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:33.124054 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:33.124069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.126657 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.126666 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:33.129541 (+ 2875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:33.129554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.132351 (+ 2797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.132363 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:33.132661 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.132724 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 12:56:33.132828 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 12:56:33.132977 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14834,"thread_start_us":9477,"threads_started":1}
I20260808 12:56:33.136466 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.106731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.24.167.2:35917)
0808 12:56:33.119326 (+ 12595us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:33.119339 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:33.120965 (+ 1626us) server_negotiation.cc:407] Connection header received
0808 12:56:33.121566 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:33.121573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:33.121801 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:33.122004 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:33.123877 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.125743 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.131071 (+ 5328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.134749 (+ 3678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.134908 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.135101 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:33.135919 (+ 818us) server_negotiation.cc:299] Negotiation successful
0808 12:56:33.136087 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12393,"thread_start_us":12248,"threads_started":1}
I20260808 12:56:33.136644 26657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.106017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:35917 (local address 127.0.0.1:50940)
0808 12:56:33.120826 (+ 14809us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:33.120845 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:33.121097 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:33.122242 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:33.122246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:33.122271 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:33.123616 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:33.123629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.125900 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.125907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:33.130834 (+ 4927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:33.130852 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.135416 (+ 4564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.135424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:33.135679 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.135712 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 12:56:33.135803 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 12:56:33.135926 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14648,"mutex_wait_us":3,"thread_start_us":15105,"threads_started":1}
I20260808 12:56:33.136719 26189 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3aee3f0f374156992100878bbe947a (DEFAULT_TABLE table=test-table [id=453d0a455a76482d86b53ea88a10530e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:33.138764 26189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3aee3f0f374156992100878bbe947a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:33.137602 26548 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3aee3f0f374156992100878bbe947a (DEFAULT_TABLE table=test-table [id=453d0a455a76482d86b53ea88a10530e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:33.141395 26548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3aee3f0f374156992100878bbe947a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:33.140803 26367 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3aee3f0f374156992100878bbe947a (DEFAULT_TABLE table=test-table [id=453d0a455a76482d86b53ea88a10530e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:33.142627 26367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3aee3f0f374156992100878bbe947a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:33.153362 26664 tablet_bootstrap.cc:492] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8: Bootstrap starting.
I20260808 12:56:33.158735 26664 tablet_bootstrap.cc:654] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:33.159966 26664 log.cc:824] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8: Log is configured to *not* fsync() on all Append() calls
I20260808 12:56:33.163484 26670 tablet_bootstrap.cc:492] T 2f3aee3f0f374156992100878bbe947a P 9b30f33bff564a71a3be525b71df12bf: Bootstrap starting.
I20260808 12:56:33.164889 26664 tablet_bootstrap.cc:492] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8: No bootstrap required, opened a new log
I20260808 12:56:33.165349 26664 ts_tablet_manager.cc:1397] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260808 12:56:33.165619 26668 tablet_bootstrap.cc:492] T 2f3aee3f0f374156992100878bbe947a P a2dd446b021447389fbf062471c09101: Bootstrap starting.
I20260808 12:56:33.170746 26668 tablet_bootstrap.cc:654] T 2f3aee3f0f374156992100878bbe947a P a2dd446b021447389fbf062471c09101: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:33.170913 26670 tablet_bootstrap.cc:654] T 2f3aee3f0f374156992100878bbe947a P 9b30f33bff564a71a3be525b71df12bf: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:33.172240 26668 log.cc:824] T 2f3aee3f0f374156992100878bbe947a P a2dd446b021447389fbf062471c09101: Log is configured to *not* fsync() on all Append() calls
I20260808 12:56:33.173205 26670 log.cc:824] T 2f3aee3f0f374156992100878bbe947a P 9b30f33bff564a71a3be525b71df12bf: Log is configured to *not* fsync() on all Append() calls
I20260808 12:56:33.176914 26670 tablet_bootstrap.cc:492] T 2f3aee3f0f374156992100878bbe947a P 9b30f33bff564a71a3be525b71df12bf: No bootstrap required, opened a new log
I20260808 12:56:33.177270 26670 ts_tablet_manager.cc:1397] T 2f3aee3f0f374156992100878bbe947a P 9b30f33bff564a71a3be525b71df12bf: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260808 12:56:33.177402 26668 tablet_bootstrap.cc:492] T 2f3aee3f0f374156992100878bbe947a P a2dd446b021447389fbf062471c09101: No bootstrap required, opened a new log
I20260808 12:56:33.177839 26668 ts_tablet_manager.cc:1397] T 2f3aee3f0f374156992100878bbe947a P a2dd446b021447389fbf062471c09101: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20260808 12:56:33.181038 26664 raft_consensus.cc:348] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263043659f9640c9a47733fca2fbe4e8" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44841 } } peers { permanent_uuid: "a2dd446b021447389fbf062471c09101" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35917 } } peers { permanent_uuid: "9b30f33bff564a71a3be525b71df12bf" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43407 } }
I20260808 12:56:33.181794 26664 raft_consensus.cc:374] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:33.182006 26664 raft_consensus.cc:729] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263043659f9640c9a47733fca2fbe4e8, State: Initialized, Role: FOLLOWER
I20260808 12:56:33.183005 26664 consensus_queue.cc:260] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263043659f9640c9a47733fca2fbe4e8" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44841 } } peers { permanent_uuid: "a2dd446b021447389fbf062471c09101" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35917 } } peers { permanent_uuid: "9b30f33bff564a71a3be525b71df12bf" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43407 } }
I20260808 12:56:33.190171 26675 raft_consensus.cc:482] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:56:33.190809 26675 raft_consensus.cc:504] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263043659f9640c9a47733fca2fbe4e8" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44841 } } peers { permanent_uuid: "a2dd446b021447389fbf062471c09101" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35917 } } peers { permanent_uuid: "9b30f33bff564a71a3be525b71df12bf" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43407 } }
I20260808 12:56:33.191013 26670 raft_consensus.cc:348] T 2f3aee3f0f374156992100878bbe947a P 9b30f33bff564a71a3be525b71df12bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263043659f9640c9a47733fca2fbe4e8" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44841 } } peers { permanent_uuid: "a2dd446b021447389fbf062471c09101" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35917 } } peers { permanent_uuid: "9b30f33bff564a71a3be525b71df12bf" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43407 } }
I20260808 12:56:33.191753 26670 raft_consensus.cc:374] T 2f3aee3f0f374156992100878bbe947a P 9b30f33bff564a71a3be525b71df12bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:33.191891 26670 raft_consensus.cc:729] T 2f3aee3f0f374156992100878bbe947a P 9b30f33bff564a71a3be525b71df12bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b30f33bff564a71a3be525b71df12bf, State: Initialized, Role: FOLLOWER
I20260808 12:56:33.192705 26670 consensus_queue.cc:260] T 2f3aee3f0f374156992100878bbe947a P 9b30f33bff564a71a3be525b71df12bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263043659f9640c9a47733fca2fbe4e8" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44841 } } peers { permanent_uuid: "a2dd446b021447389fbf062471c09101" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35917 } } peers { permanent_uuid: "9b30f33bff564a71a3be525b71df12bf" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43407 } }
I20260808 12:56:33.195326 26668 raft_consensus.cc:348] T 2f3aee3f0f374156992100878bbe947a P a2dd446b021447389fbf062471c09101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263043659f9640c9a47733fca2fbe4e8" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44841 } } peers { permanent_uuid: "a2dd446b021447389fbf062471c09101" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35917 } } peers { permanent_uuid: "9b30f33bff564a71a3be525b71df12bf" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43407 } }
I20260808 12:56:33.195888 26668 raft_consensus.cc:374] T 2f3aee3f0f374156992100878bbe947a P a2dd446b021447389fbf062471c09101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:33.196060 26668 raft_consensus.cc:729] T 2f3aee3f0f374156992100878bbe947a P a2dd446b021447389fbf062471c09101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2dd446b021447389fbf062471c09101, State: Initialized, Role: FOLLOWER
I20260808 12:56:33.196897 26668 consensus_queue.cc:260] T 2f3aee3f0f374156992100878bbe947a P a2dd446b021447389fbf062471c09101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263043659f9640c9a47733fca2fbe4e8" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44841 } } peers { permanent_uuid: "a2dd446b021447389fbf062471c09101" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35917 } } peers { permanent_uuid: "9b30f33bff564a71a3be525b71df12bf" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43407 } }
I20260808 12:56:33.203745 26670 ts_tablet_manager.cc:1428] T 2f3aee3f0f374156992100878bbe947a P 9b30f33bff564a71a3be525b71df12bf: Time spent starting tablet: real 0.026s user 0.023s sys 0.004s
I20260808 12:56:33.204061 26664 ts_tablet_manager.cc:1428] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8: Time spent starting tablet: real 0.039s user 0.017s sys 0.011s
I20260808 12:56:33.204442 26638 heartbeater.cc:503] Master 127.24.167.62:42525 was elected leader, sending a full tablet report...
I20260808 12:56:33.205389 26675 leader_election.cc:288] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2dd446b021447389fbf062471c09101 (127.24.167.2:35917), 9b30f33bff564a71a3be525b71df12bf (127.24.167.3:43407)
I20260808 12:56:33.211766 26459 heartbeater.cc:503] Master 127.24.167.62:42525 was elected leader, sending a full tablet report...
I20260808 12:56:33.212580 26668 ts_tablet_manager.cc:1428] T 2f3aee3f0f374156992100878bbe947a P a2dd446b021447389fbf062471c09101: Time spent starting tablet: real 0.035s user 0.005s sys 0.028s
I20260808 12:56:33.216493 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.195446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:52083 (local address 127.24.167.2:35917)
0808 12:56:33.195744 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:33.195753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:33.205373 (+ 9620us) server_negotiation.cc:407] Connection header received
0808 12:56:33.205773 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:33.205782 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:33.205890 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:33.205971 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:33.208137 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.209805 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.213797 (+ 3992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.215221 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.215375 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.215447 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:33.216032 (+ 585us) server_negotiation.cc:299] Negotiation successful
0808 12:56:33.216211 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 12:56:33.216995 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.194756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:35917 (local address 127.24.167.1:52083)
0808 12:56:33.205222 (+ 10466us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:33.205248 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:33.205539 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:33.206859 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:33.206866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:33.206905 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:33.207791 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:33.207808 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.210046 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.210054 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:33.213547 (+ 3493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:33.213567 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.215574 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.215582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:33.215848 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.215884 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 12:56:33.215969 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 12:56:33.216144 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10253,"thread_start_us":17166,"threads_started":1}
I20260808 12:56:33.218586 26392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3aee3f0f374156992100878bbe947a" candidate_uuid: "263043659f9640c9a47733fca2fbe4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2dd446b021447389fbf062471c09101" is_pre_election: true
I20260808 12:56:33.219298 26392 raft_consensus.cc:2454] T 2f3aee3f0f374156992100878bbe947a P a2dd446b021447389fbf062471c09101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 263043659f9640c9a47733fca2fbe4e8 in term 0.
I20260808 12:56:33.220391 26130 leader_election.cc:302] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263043659f9640c9a47733fca2fbe4e8, a2dd446b021447389fbf062471c09101; no voters:
I20260808 12:56:33.220911 26675 raft_consensus.cc:2781] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:56:33.221139 26675 raft_consensus.cc:482] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:56:33.221376 26675 raft_consensus.cc:3037] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:33.222690 26663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.201734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:33069 (local address 127.24.167.3:43407)
0808 12:56:33.202123 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:33.202132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:33.212548 (+ 10416us) server_negotiation.cc:407] Connection header received
0808 12:56:33.213044 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:33.213050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:33.213141 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:33.213260 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:33.214302 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.215872 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.219959 (+ 4087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.221137 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.221316 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.221393 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:33.222123 (+ 730us) server_negotiation.cc:299] Negotiation successful
0808 12:56:33.222296 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 12:56:33.223727 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.202725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:43407 (local address 127.24.167.1:33069)
0808 12:56:33.212445 (+ 9720us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:33.212466 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:33.212785 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:33.213286 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:33.213290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:33.213318 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:33.214053 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:33.214071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.216081 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.216086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:33.219643 (+ 3557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:33.219659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.221540 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.221548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:33.221864 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.221921 (+ 57us) client_negotiation.cc:769] Sending connection context
0808 12:56:33.222929 (+ 1008us) client_negotiation.cc:240] Negotiation successful
0808 12:56:33.223091 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8996,"thread_start_us":2008,"threads_started":1}
I20260808 12:56:33.224946 26578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3aee3f0f374156992100878bbe947a" candidate_uuid: "263043659f9640c9a47733fca2fbe4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b30f33bff564a71a3be525b71df12bf" is_pre_election: true
I20260808 12:56:33.225455 26675 raft_consensus.cc:504] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263043659f9640c9a47733fca2fbe4e8" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44841 } } peers { permanent_uuid: "a2dd446b021447389fbf062471c09101" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35917 } } peers { permanent_uuid: "9b30f33bff564a71a3be525b71df12bf" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43407 } }
I20260808 12:56:33.225728 26578 raft_consensus.cc:2454] T 2f3aee3f0f374156992100878bbe947a P 9b30f33bff564a71a3be525b71df12bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 263043659f9640c9a47733fca2fbe4e8 in term 0.
I20260808 12:56:33.226163 26675 leader_election.cc:288] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [CANDIDATE]: Term 1 election: Requested vote from peers a2dd446b021447389fbf062471c09101 (127.24.167.2:35917), 9b30f33bff564a71a3be525b71df12bf (127.24.167.3:43407)
I20260808 12:56:33.226766 26392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3aee3f0f374156992100878bbe947a" candidate_uuid: "263043659f9640c9a47733fca2fbe4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2dd446b021447389fbf062471c09101"
I20260808 12:56:33.226763 26578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3aee3f0f374156992100878bbe947a" candidate_uuid: "263043659f9640c9a47733fca2fbe4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b30f33bff564a71a3be525b71df12bf"
I20260808 12:56:33.227010 26578 raft_consensus.cc:3037] T 2f3aee3f0f374156992100878bbe947a P 9b30f33bff564a71a3be525b71df12bf [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:33.227044 26392 raft_consensus.cc:3037] T 2f3aee3f0f374156992100878bbe947a P a2dd446b021447389fbf062471c09101 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:33.229887 26578 raft_consensus.cc:2454] T 2f3aee3f0f374156992100878bbe947a P 9b30f33bff564a71a3be525b71df12bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263043659f9640c9a47733fca2fbe4e8 in term 1.
I20260808 12:56:33.229861 26392 raft_consensus.cc:2454] T 2f3aee3f0f374156992100878bbe947a P a2dd446b021447389fbf062471c09101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263043659f9640c9a47733fca2fbe4e8 in term 1.
I20260808 12:56:33.230337 26130 leader_election.cc:302] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263043659f9640c9a47733fca2fbe4e8, a2dd446b021447389fbf062471c09101; no voters:
I20260808 12:56:33.230690 26675 raft_consensus.cc:2781] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:33.230886 26675 raft_consensus.cc:686] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [term 1 LEADER]: Becoming Leader. State: Replica: 263043659f9640c9a47733fca2fbe4e8, State: Running, Role: LEADER
I20260808 12:56:33.231381 26675 consensus_queue.cc:237] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263043659f9640c9a47733fca2fbe4e8" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44841 } } peers { permanent_uuid: "a2dd446b021447389fbf062471c09101" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35917 } } peers { permanent_uuid: "9b30f33bff564a71a3be525b71df12bf" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43407 } }
I20260808 12:56:33.236389 26019 catalog_manager.cc:5495] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 263043659f9640c9a47733fca2fbe4e8 (127.24.167.1). New cstate: current_term: 1 leader_uuid: "263043659f9640c9a47733fca2fbe4e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263043659f9640c9a47733fca2fbe4e8" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2dd446b021447389fbf062471c09101" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b30f33bff564a71a3be525b71df12bf" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43407 } health_report { overall_health: UNKNOWN } } }
W20260808 12:56:33.258318 26640 tablet.cc:2369] T 2f3aee3f0f374156992100878bbe947a P 9b30f33bff564a71a3be525b71df12bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 12:56:33 dist-test-slave-tk9z krb5kdc[25957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:56:33.270437 26660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.250351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.24.167.1:44841)
0808 12:56:33.250755 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:33.250764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:33.251584 (+ 820us) server_negotiation.cc:407] Connection header received
0808 12:56:33.251756 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:33.251761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:33.251838 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:33.252002 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:33.254132 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.255151 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.256608 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.257133 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.257233 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.262944 (+ 5711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:33.262961 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:33.262977 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:33.263012 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:33.267048 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.267981 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.267991 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.267995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.268217 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.268724 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.268728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.268730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.269153 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:33.269406 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:33.269731 (+ 325us) server_negotiation.cc:299] Negotiation successful
0808 12:56:33.269911 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
Aug 08 12:56:33 dist-test-slave-tk9z krb5kdc[25957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193793, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 12:56:33 dist-test-slave-tk9z krb5kdc[25957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193793, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:56:33.327351 26660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.309299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.24.167.1:44841)
0808 12:56:33.309517 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:33.309525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:33.310128 (+ 603us) server_negotiation.cc:407] Connection header received
0808 12:56:33.310361 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:33.310409 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:33.310490 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:33.310602 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:33.312740 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.313528 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.314970 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.315675 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.315833 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.320884 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:33.320889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:33.320892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:33.320931 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:33.324461 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.325107 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.325117 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.325120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.325295 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.325713 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.325719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.325722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.326085 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:33.326605 (+ 520us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:33.326729 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 12:56:33.326997 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 12:56:33 dist-test-slave-tk9z krb5kdc[25957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 12:56:33 dist-test-slave-tk9z krb5kdc[25957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:56:33.382215 26660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.364903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52234 (local address 127.24.167.1:44841)
0808 12:56:33.365192 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:33.365200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:33.365671 (+ 471us) server_negotiation.cc:407] Connection header received
0808 12:56:33.365908 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:33.365912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:33.366001 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:33.366116 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:33.368005 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.368964 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.370439 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.370956 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.371104 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.375775 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:33.375780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:33.375784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:33.375825 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:33.379239 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.380022 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.380029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.380033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.380245 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.380629 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.380634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.380637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.380967 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:33.381245 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:33.381589 (+ 344us) server_negotiation.cc:299] Negotiation successful
0808 12:56:33.381817 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
W20260808 12:56:33.384370 26164 tablet_service.cc:2908] Rejecting scan request for tablet 2f3aee3f0f374156992100878bbe947a: Uninitialized: clean time has not yet been initialized
I20260808 12:56:33.387876 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26107
I20260808 12:56:33.403717 26660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.388934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52248 (local address 127.24.167.1:44841)
0808 12:56:33.389100 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:33.389105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:33.389566 (+ 461us) server_negotiation.cc:407] Connection header received
0808 12:56:33.389725 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:33.389729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:33.389796 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:33.390084 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:33.391412 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.392474 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.393681 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.394273 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.394420 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.397053 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:33.397057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:33.397060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:33.397092 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:33.400593 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.401394 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.401397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.401400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.401589 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.402031 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.402034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.402036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.402516 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:33.402799 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:33.403033 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 12:56:33.403327 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 12:56:33.404587 26271 generic_service.cc:196] Checking for leaks (request via RPC)
W20260808 12:56:33.458715 26460 tablet.cc:2369] T 2f3aee3f0f374156992100878bbe947a P a2dd446b021447389fbf062471c09101: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 12:56:33.499536 26281 tablet.cc:2369] T 2f3aee3f0f374156992100878bbe947a P 263043659f9640c9a47733fca2fbe4e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:56:33.500418 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 26107
I20260808 12:56:33.525362 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26286
Aug 08 12:56:33 dist-test-slave-tk9z krb5kdc[25957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 12:56:33.541653 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.526250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.24.167.2:35917)
0808 12:56:33.526501 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:33.526507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:33.526529 (+ 22us) server_negotiation.cc:407] Connection header received
0808 12:56:33.526609 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:33.526613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:33.526674 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:33.526801 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:33.528153 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.528934 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.530206 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.530693 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.530827 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.535503 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:33.535518 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:33.535534 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:33.535567 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:33.539130 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.539774 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.539782 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.539784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.539909 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.540233 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.540236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.540238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.540603 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:33.540846 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:33.541104 (+ 258us) server_negotiation.cc:299] Negotiation successful
0808 12:56:33.541272 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 12:56:33.542424 26452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:56:33.631426 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 26286
I20260808 12:56:33.659883 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26466
Aug 08 12:56:33 dist-test-slave-tk9z krb5kdc[25957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 12:56:33.675940 26663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:33.660756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.24.167.3:43407)
0808 12:56:33.660958 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:33.660962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:33.660979 (+ 17us) server_negotiation.cc:407] Connection header received
0808 12:56:33.661040 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:33.661045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:33.661110 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:33.661202 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:33.662555 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.663426 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.664765 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:33.665183 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:33.665284 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:33.670135 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:33.670153 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:33.670173 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:33.670213 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:33.673232 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.673946 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.673955 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.673957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.674096 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:33.674504 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:33.674508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:33.674510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:33.674941 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:33.675206 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:33.675476 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 12:56:33.675665 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 12:56:33.676653 26632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:56:33.766772 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 26466
I20260808 12:56:33.790730 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 25975
I20260808 12:56:33.791915 26061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:56:33.904053 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 25975
2026-08-08T12:56:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4528 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:56:34 dist-test-slave-tk9z krb5kdc[26702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:56:34 dist-test-slave-tk9z krb5kdc[26702](info): set up 2 sockets
Aug 08 12:56:34 dist-test-slave-tk9z krb5kdc[26702](info): commencing operation
krb5kdc: starting...
W20260808 12:56:35.212033 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.257s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:56:35 dist-test-slave-tk9z krb5kdc[26702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:56:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:56:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 12:56:35.406262 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:42591
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:42591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:35.749395 26715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:35.749742 26715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:35.749917 26715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:35.758037 26715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:56:35.758152 26715 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 12:56:35.758252 26715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:35.758399 26715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:35.758489 26715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:56:35.758608 26715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:56:35.769562 26715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:42591
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:42591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:35.771061 26715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:35.772496 26715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:35.784734 26721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:35.784603 26722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:35.784634 26726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:35.786060 26715 server_base.cc:1034] running on GCE node
I20260808 12:56:35.786954 26715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:35.788304 26715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:35.789533 26715 hybrid_clock.cc:648] HybridClock initialized: now 1786193795789477 us; error 28 us; skew 500 ppm
I20260808 12:56:35.789870 26715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:35 dist-test-slave-tk9z krb5kdc[26702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:35.796789 26715 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:56:35.798246 26715 webserver.cc:460] Webserver started at http://127.24.167.62:38913/ using document root <none> and password file <none>
I20260808 12:56:35.799014 26715 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:35.799104 26715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:35.799373 26715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:35.802306 26715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "3fa10f97aad34da198e63292d9dbb90c"
format_stamp: "Formatted at 2026-08-08 12:56:35 on dist-test-slave-tk9z"
server_key: "5652e0423e67a3635b69b51fe395ecda"
server_key_iv: "c8bf0d680ea34a0be9efc635c4872203"
server_key_version: "encryptionkey@0"
I20260808 12:56:35.803117 26715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "3fa10f97aad34da198e63292d9dbb90c"
format_stamp: "Formatted at 2026-08-08 12:56:35 on dist-test-slave-tk9z"
server_key: "5652e0423e67a3635b69b51fe395ecda"
server_key_iv: "c8bf0d680ea34a0be9efc635c4872203"
server_key_version: "encryptionkey@0"
I20260808 12:56:35.808324 26715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 12:56:35.811703 26741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:35.813032 26715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 12:56:35.813231 26715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "3fa10f97aad34da198e63292d9dbb90c"
format_stamp: "Formatted at 2026-08-08 12:56:35 on dist-test-slave-tk9z"
server_key: "5652e0423e67a3635b69b51fe395ecda"
server_key_iv: "c8bf0d680ea34a0be9efc635c4872203"
server_key_version: "encryptionkey@0"
I20260808 12:56:35.813450 26715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:35.849965 26715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:35.855038 26715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:35.855404 26715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:35.879907 26715 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:42591
I20260808 12:56:35.879944 26821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:42591 every 8 connection(s)
I20260808 12:56:35.881466 26715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:56:35.885311 26822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:35.886333 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 26715
I20260808 12:56:35.886637 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:56:35.887315 25244 external_mini_cluster.cc:1428] Setting key 7c78ca68144d894971439f35c9bfc6f0
I20260808 12:56:35.900141 26822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c: Bootstrap starting.
Aug 08 12:56:35 dist-test-slave-tk9z krb5kdc[26702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:35.903355 26822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:35.904742 26822 log.cc:824] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c: Log is configured to *not* fsync() on all Append() calls
I20260808 12:56:35.908140 26822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c: No bootstrap required, opened a new log
I20260808 12:56:35.911645 26826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:35.889357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47740 (local address 127.24.167.62:42591)
0808 12:56:35.890232 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:35.890248 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:35.890312 (+ 64us) server_negotiation.cc:407] Connection header received
0808 12:56:35.892250 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:35.892278 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:35.892815 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:35.893301 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:35.894595 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:35.896654 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:35.897931 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:35.898985 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:35.899163 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:35.903205 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:35.903218 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:35.903244 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:35.903309 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:35.907134 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:35.907780 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:35.907788 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:35.907795 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:35.908001 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:35.908384 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:35.908389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:35.908392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:35.909131 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:35.909424 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:35.909725 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 12:56:35.910180 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":344,"threads_started":1}
I20260808 12:56:35.917476 26822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa10f97aad34da198e63292d9dbb90c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42591 } }
I20260808 12:56:35.917837 26822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:35.917966 26822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa10f97aad34da198e63292d9dbb90c, State: Initialized, Role: FOLLOWER
I20260808 12:56:35.918670 26822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa10f97aad34da198e63292d9dbb90c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42591 } }
I20260808 12:56:35.918972 26822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:56:35.919137 26822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:56:35.919307 26822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:35.921502 26822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa10f97aad34da198e63292d9dbb90c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42591 } }
I20260808 12:56:35.922020 26822 leader_election.cc:302] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fa10f97aad34da198e63292d9dbb90c; no voters:
I20260808 12:56:35.922705 26822 leader_election.cc:288] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:56:35.922819 26828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:35.923771 26828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [term 1 LEADER]: Becoming Leader. State: Replica: 3fa10f97aad34da198e63292d9dbb90c, State: Running, Role: LEADER
I20260808 12:56:35.924360 26828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa10f97aad34da198e63292d9dbb90c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42591 } }
I20260808 12:56:35.925274 26822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:56:35.927621 26829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fa10f97aad34da198e63292d9dbb90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa10f97aad34da198e63292d9dbb90c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42591 } } }
I20260808 12:56:35.928056 26829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [sys.catalog]: This master's current role is: LEADER
I20260808 12:56:35.927605 26830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fa10f97aad34da198e63292d9dbb90c. Latest consensus state: current_term: 1 leader_uuid: "3fa10f97aad34da198e63292d9dbb90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa10f97aad34da198e63292d9dbb90c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42591 } } }
I20260808 12:56:35.928720 26830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c [sys.catalog]: This master's current role is: LEADER
I20260808 12:56:35.931574 26833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:56:35.936816 26833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:56:35.947415 26833 catalog_manager.cc:1347] Generated new cluster ID: 902ad19cea074469b5e73778ca31821b
I20260808 12:56:35.947553 26833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:56:35.958806 26833 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:56:35.960638 26833 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:56:35.975850 26833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c: Generated new TSK 0
I20260808 12:56:35.976965 26833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 12:56:36.037693 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42591
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:36.380589 26851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:36.380910 26851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:36.381084 26851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:36.388909 26851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:36.389088 26851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:36.389235 26851 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 12:56:36.389372 26851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:56:36.399750 26851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.167.62:42591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:36.401160 26851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:36.402645 26851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:36.416059 26856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:36.416100 26860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:36.416024 26857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:36.416954 26851 server_base.cc:1034] running on GCE node
I20260808 12:56:36.417719 26851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:36.418725 26851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:36.419945 26851 hybrid_clock.cc:648] HybridClock initialized: now 1786193796419870 us; error 46 us; skew 500 ppm
I20260808 12:56:36.420252 26851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:36 dist-test-slave-tk9z krb5kdc[26702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:36.423692 26851 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 12:56:36.425202 26851 webserver.cc:460] Webserver started at http://127.24.167.1:38751/ using document root <none> and password file <none>
I20260808 12:56:36.425967 26851 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:36.426095 26851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:36.426548 26851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:36.429637 26851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "e30b4e9353ad4d07b0d47069d25327d7"
format_stamp: "Formatted at 2026-08-08 12:56:36 on dist-test-slave-tk9z"
server_key: "10eb89723639178c3cf866d03cd681d3"
server_key_iv: "176f85bc8bcdfb80b618bf0e71f0ce4c"
server_key_version: "encryptionkey@0"
I20260808 12:56:36.430546 26851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "e30b4e9353ad4d07b0d47069d25327d7"
format_stamp: "Formatted at 2026-08-08 12:56:36 on dist-test-slave-tk9z"
server_key: "10eb89723639178c3cf866d03cd681d3"
server_key_iv: "176f85bc8bcdfb80b618bf0e71f0ce4c"
server_key_version: "encryptionkey@0"
I20260808 12:56:36.435853 26851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 12:56:36.439333 26869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:36.440744 26851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 12:56:36.440958 26851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "e30b4e9353ad4d07b0d47069d25327d7"
format_stamp: "Formatted at 2026-08-08 12:56:36 on dist-test-slave-tk9z"
server_key: "10eb89723639178c3cf866d03cd681d3"
server_key_iv: "176f85bc8bcdfb80b618bf0e71f0ce4c"
server_key_version: "encryptionkey@0"
I20260808 12:56:36.441280 26851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:36.460131 26851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:36.464184 26851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:36.464476 26851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:36.467447 26851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:36.467554 26851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:36.467733 26851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:36.467808 26851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 12:56:36 dist-test-slave-tk9z krb5kdc[26702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:36.491966 26880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:36.467925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:57793 (local address 127.24.167.62:42591)
0808 12:56:36.468746 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:36.468751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:36.469598 (+ 847us) server_negotiation.cc:407] Connection header received
0808 12:56:36.471723 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:36.471730 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:36.471817 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:36.471934 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:36.474162 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:36.475474 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:36.478082 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:36.478877 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:36.479008 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:36.484695 (+ 5687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:36.484701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:36.484704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:36.484740 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:36.487957 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:36.489066 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:36.489073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:36.489076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:36.489239 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:36.489954 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:36.489958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:36.489960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:36.490236 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:36.490363 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:36.491534 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0808 12:56:36.491706 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":624,"threads_started":1}
I20260808 12:56:36.494138 26881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:36.468595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42591 (local address 127.24.167.1:57793)
0808 12:56:36.469433 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:36.469477 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:36.471415 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:36.472176 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:36.472184 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:36.472984 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:36.473923 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:36.473952 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:36.475694 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:36.475698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:36.477914 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:36.477933 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:36.479153 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:36.479167 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:36.479364 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:36.480150 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:36.480211 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:36.484397 (+ 4186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:36.488151 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:36.488161 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:36.488177 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:36.488827 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:36.489365 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:36.489370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:36.489374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:36.489784 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:36.490451 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:36.490464 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:36.491068 (+ 604us) client_negotiation.cc:769] Sending connection context
0808 12:56:36.491470 (+ 402us) client_negotiation.cc:240] Negotiation successful
0808 12:56:36.492203 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":366,"threads_started":1}
I20260808 12:56:36.516427 27030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:38983 every 8 connection(s)
I20260808 12:56:36.516462 26851 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:38983
I20260808 12:56:36.518286 26851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:56:36.519047 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 26851
I20260808 12:56:36.519286 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:56:36.520143 25244 external_mini_cluster.cc:1428] Setting key 3ac1a3581c133da616d24cfa16fcabf9
I20260808 12:56:36.520226 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:36.503156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42591 (local address 127.24.167.1:48493)
0808 12:56:36.504166 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:36.504192 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:36.504374 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:36.504851 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:36.504856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:36.505344 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:36.505652 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:36.505661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:36.507259 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:36.507263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:36.508667 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:36.508681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:36.509816 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:36.509821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:36.509951 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:36.510742 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:36.510773 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:36.512245 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:36.516653 (+ 4408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:36.516658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:36.516663 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:36.517381 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:36.517862 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:36.517866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:36.517869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:36.518085 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:36.518860 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:36.518867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:36.519104 (+ 237us) client_negotiation.cc:769] Sending connection context
0808 12:56:36.519314 (+ 210us) client_negotiation.cc:240] Negotiation successful
0808 12:56:36.519709 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":618,"threads_started":1}
I20260808 12:56:36.520778 26880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:36.502991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:48493 (local address 127.24.167.62:42591)
0808 12:56:36.503198 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:36.503205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:36.504311 (+ 1106us) server_negotiation.cc:407] Connection header received
0808 12:56:36.504521 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:36.504530 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:36.504644 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:36.504784 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:36.505809 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:36.507101 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:36.508825 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:36.509614 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:36.509765 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:36.512507 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:36.512516 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:36.512524 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:36.512571 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:36.516508 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:36.517514 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:36.517520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:36.517523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:36.517691 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:36.518227 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:36.518235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:36.518258 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:36.518700 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:36.518883 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:36.519555 (+ 672us) server_negotiation.cc:299] Negotiation successful
0808 12:56:36.519887 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 12:56:36.524580 27031 heartbeater.cc:347] Connected to a master server at 127.24.167.62:42591
I20260808 12:56:36.524899 27031 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:36.526057 27031 heartbeater.cc:511] Master 127.24.167.62:42591 requested a full tablet report, sending...
I20260808 12:56:36.528517 26761 ts_manager.cc:194] Registered new tserver with Master: e30b4e9353ad4d07b0d47069d25327d7 (127.24.167.1:38983)
I20260808 12:56:36.530028 26761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:57793
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 12:56:36.565804 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42591
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:36.901373 27035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:36.901657 27035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:36.901823 27035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:36.911994 27035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:36.912181 27035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:36.912325 27035 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 12:56:36.912566 27035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:56:36.924194 27035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.167.62:42591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:36.925582 27035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:36.927078 27035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:36.940577 27052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:36.940478 27053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:36.940515 27055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:36.941308 27035 server_base.cc:1034] running on GCE node
I20260808 12:56:36.941917 27035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:36.942910 27035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:36.944160 27035 hybrid_clock.cc:648] HybridClock initialized: now 1786193796944080 us; error 55 us; skew 500 ppm
I20260808 12:56:36.944471 27035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:36 dist-test-slave-tk9z krb5kdc[26702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:36.947943 27035 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 12:56:36.949360 27035 webserver.cc:460] Webserver started at http://127.24.167.2:33129/ using document root <none> and password file <none>
I20260808 12:56:36.950057 27035 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:36.950189 27035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:36.950593 27035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:36.953469 27035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "89a412042609494cb20203ee62d34dfd"
format_stamp: "Formatted at 2026-08-08 12:56:36 on dist-test-slave-tk9z"
server_key: "5dbb5898f968a31bcd16f3643a26b9d4"
server_key_iv: "37048fd2a59bf15abec7e398f27e2ad5"
server_key_version: "encryptionkey@0"
I20260808 12:56:36.954277 27035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "89a412042609494cb20203ee62d34dfd"
format_stamp: "Formatted at 2026-08-08 12:56:36 on dist-test-slave-tk9z"
server_key: "5dbb5898f968a31bcd16f3643a26b9d4"
server_key_iv: "37048fd2a59bf15abec7e398f27e2ad5"
server_key_version: "encryptionkey@0"
I20260808 12:56:36.959340 27035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 12:56:36.962852 27063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:36.964202 27035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 12:56:36.964393 27035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "89a412042609494cb20203ee62d34dfd"
format_stamp: "Formatted at 2026-08-08 12:56:36 on dist-test-slave-tk9z"
server_key: "5dbb5898f968a31bcd16f3643a26b9d4"
server_key_iv: "37048fd2a59bf15abec7e398f27e2ad5"
server_key_version: "encryptionkey@0"
I20260808 12:56:36.964622 27035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:37.002761 27035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:37.008158 27035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:37.008420 27035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:37.011502 27035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:37.011626 27035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:37.011809 27035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:37.011881 27035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 12:56:37 dist-test-slave-tk9z krb5kdc[26702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:37.040673 26880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.012016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:35275 (local address 127.24.167.62:42591)
0808 12:56:37.012215 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:37.012224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:37.013725 (+ 1501us) server_negotiation.cc:407] Connection header received
0808 12:56:37.015871 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:37.015877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:37.015983 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:37.016100 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:37.018853 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.019932 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.023709 (+ 3777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.024483 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.024611 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.030832 (+ 6221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:37.030849 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:37.030855 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:37.030911 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:37.035382 (+ 4471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:37.036794 (+ 1412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:37.036802 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:37.036807 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:37.037014 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:37.037765 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:37.037773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:37.037779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:37.038286 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:37.038520 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:37.039839 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0808 12:56:37.040160 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 12:56:37.042644 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.012745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42591 (local address 127.24.167.2:35275)
0808 12:56:37.013557 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:37.013596 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:37.015569 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:37.016353 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:37.016361 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:37.017303 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:37.018582 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.018616 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.020126 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.020134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.023530 (+ 3396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.023545 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.024802 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.024812 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.025211 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.026194 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:37.026234 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:37.030503 (+ 4269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:37.035613 (+ 5110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:37.035628 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:37.035658 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:37.036548 (+ 890us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:37.037169 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:37.037179 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:37.037187 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:37.037600 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:37.038497 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:37.038506 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:37.039196 (+ 690us) client_negotiation.cc:769] Sending connection context
0808 12:56:37.039688 (+ 492us) client_negotiation.cc:240] Negotiation successful
0808 12:56:37.040475 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":354,"threads_started":1}
I20260808 12:56:37.057355 27035 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:39019
I20260808 12:56:37.057363 27224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:39019 every 8 connection(s)
I20260808 12:56:37.058915 27035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:56:37.061316 27226 heartbeater.cc:347] Connected to a master server at 127.24.167.62:42591
I20260808 12:56:37.061606 27226 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:37.062566 27226 heartbeater.cc:511] Master 127.24.167.62:42591 requested a full tablet report, sending...
I20260808 12:56:37.064296 26761 ts_manager.cc:194] Registered new tserver with Master: 89a412042609494cb20203ee62d34dfd (127.24.167.2:39019)
I20260808 12:56:37.065382 26761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:35275
I20260808 12:56:37.066689 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 27035
I20260808 12:56:37.066887 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:56:37.067039 26880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.049652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:56349 (local address 127.24.167.62:42591)
0808 12:56:37.049867 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:37.049876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:37.050798 (+ 922us) server_negotiation.cc:407] Connection header received
0808 12:56:37.051116 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:37.051121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:37.051193 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:37.051314 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:37.052743 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.054130 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.056565 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.057299 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.057453 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.060112 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:37.060119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:37.060124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:37.060173 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:37.063936 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:37.064729 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:37.064733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:37.064736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:37.064891 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:37.065287 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:37.065291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:37.065293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:37.065694 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:37.065862 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:37.066462 (+ 600us) server_negotiation.cc:299] Negotiation successful
0808 12:56:37.066698 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 12:56:37.067159 27197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.049788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42591 (local address 127.24.167.2:56349)
0808 12:56:37.050675 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:37.050707 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:37.050947 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:37.051362 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:37.051369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:37.052039 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:37.052552 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.052576 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.054342 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.054351 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.056319 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.056339 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.057528 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.057540 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.057761 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.058570 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:37.058589 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:37.059824 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:37.064075 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:37.064080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:37.064083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:37.064602 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:37.065009 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:37.065016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:37.065020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:37.065175 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:37.065841 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:37.065847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:37.066065 (+ 218us) client_negotiation.cc:769] Sending connection context
0808 12:56:37.066230 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 12:56:37.066637 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":464,"threads_started":1}
I20260808 12:56:37.067467 25244 external_mini_cluster.cc:1428] Setting key 779172b2d3428931e73cd94e100c93fe
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 12:56:37.113587 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42591
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:37.501236 27232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:37.501621 27232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:37.501740 27232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:37.510582 27232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:37.510720 27232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:37.510859 27232 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 12:56:37.510960 27232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:56:37.523027 27232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.167.62:42591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:37.524473 27232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:37.525972 27232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 12:56:37.533162 27031 heartbeater.cc:503] Master 127.24.167.62:42591 was elected leader, sending a full tablet report...
W20260808 12:56:37.541805 27238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:37.541836 27239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:37.542424 27242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:37.542816 27232 server_base.cc:1034] running on GCE node
I20260808 12:56:37.543668 27232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:37.544797 27232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:37.546126 27232 hybrid_clock.cc:648] HybridClock initialized: now 1786193797546037 us; error 58 us; skew 500 ppm
I20260808 12:56:37.546528 27232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:37 dist-test-slave-tk9z krb5kdc[26702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193797, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:37.550071 27232 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 12:56:37.551834 27232 webserver.cc:460] Webserver started at http://127.24.167.3:32895/ using document root <none> and password file <none>
I20260808 12:56:37.552582 27232 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:37.552683 27232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:37.553004 27232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:37.556244 27232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "8f037916297f43c3a388aaeec0691b02"
format_stamp: "Formatted at 2026-08-08 12:56:37 on dist-test-slave-tk9z"
server_key: "e8c5fdbf4bb17785df923a283a05bae3"
server_key_iv: "05d7450467e42d99866fad1e366e744c"
server_key_version: "encryptionkey@0"
I20260808 12:56:37.557086 27232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "8f037916297f43c3a388aaeec0691b02"
format_stamp: "Formatted at 2026-08-08 12:56:37 on dist-test-slave-tk9z"
server_key: "e8c5fdbf4bb17785df923a283a05bae3"
server_key_iv: "05d7450467e42d99866fad1e366e744c"
server_key_version: "encryptionkey@0"
I20260808 12:56:37.562819 27232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 12:56:37.566923 27253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:37.568470 27232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 12:56:37.568622 27232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "8f037916297f43c3a388aaeec0691b02"
format_stamp: "Formatted at 2026-08-08 12:56:37 on dist-test-slave-tk9z"
server_key: "e8c5fdbf4bb17785df923a283a05bae3"
server_key_iv: "05d7450467e42d99866fad1e366e744c"
server_key_version: "encryptionkey@0"
I20260808 12:56:37.568871 27232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:37.580673 27232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:37.585402 27232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:37.585698 27232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:37.588546 27232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:37.588660 27232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:37.588807 27232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:37.588881 27232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 12:56:37 dist-test-slave-tk9z krb5kdc[26702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193797, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:37.614277 27264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.589327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:33225 (local address 127.24.167.62:42591)
0808 12:56:37.589802 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:37.589807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:37.591038 (+ 1231us) server_negotiation.cc:407] Connection header received
0808 12:56:37.593036 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:37.593043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:37.593143 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:37.593276 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:37.596050 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.597226 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.600111 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.600859 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.601016 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.606734 (+ 5718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:37.606739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:37.606743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:37.606783 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:37.610578 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:37.611364 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:37.611370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:37.611373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:37.611548 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:37.612211 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:37.612214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:37.612216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:37.612533 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:37.612717 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:37.613815 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0808 12:56:37.613993 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":268,"threads_started":1}
I20260808 12:56:37.616441 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.590008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42591 (local address 127.24.167.3:33225)
0808 12:56:37.590860 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:37.590898 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:37.592744 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:37.593571 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:37.593585 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:37.594706 (+ 1121us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:37.595796 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.595843 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.597389 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.597395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.599904 (+ 2509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.599937 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.601136 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.601171 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.601497 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.602464 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:37.602488 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:37.606405 (+ 3917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:37.610747 (+ 4342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:37.610757 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:37.610776 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:37.611225 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:37.611693 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:37.611701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:37.611708 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:37.612073 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:37.612692 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:37.612704 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:37.613314 (+ 610us) client_negotiation.cc:769] Sending connection context
0808 12:56:37.613711 (+ 397us) client_negotiation.cc:240] Negotiation successful
0808 12:56:37.614476 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":397,"threads_started":1}
I20260808 12:56:37.642598 27232 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:37555
I20260808 12:56:37.642656 27412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:37555 every 8 connection(s)
I20260808 12:56:37.643002 27264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.624572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:33287 (local address 127.24.167.62:42591)
0808 12:56:37.624850 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:37.624858 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:37.626527 (+ 1669us) server_negotiation.cc:407] Connection header received
0808 12:56:37.626845 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:37.626855 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:37.626956 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:37.627090 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:37.628761 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.630097 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.631673 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.632493 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.632641 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.635333 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:37.635340 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:37.635345 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:37.635389 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:37.639354 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:37.640449 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:37.640454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:37.640456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:37.640586 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:37.640995 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:37.641001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:37.641004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:37.641467 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:37.641642 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:37.642279 (+ 637us) server_negotiation.cc:299] Negotiation successful
0808 12:56:37.642618 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 12:56:37.643131 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.625382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42591 (local address 127.24.167.3:33287)
0808 12:56:37.626387 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:37.626414 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:37.626606 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:37.627141 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:37.627152 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:37.627930 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:37.628541 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.628562 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.630302 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.630307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.631513 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.631526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.632701 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.632706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.632836 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.633566 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:37.633602 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:37.635112 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:37.639567 (+ 4455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:37.639572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:37.639580 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:37.640306 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:37.640708 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:37.640714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:37.640719 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:37.640871 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:37.641615 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:37.641627 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:37.641891 (+ 264us) client_negotiation.cc:769] Sending connection context
0808 12:56:37.642084 (+ 193us) client_negotiation.cc:240] Negotiation successful
0808 12:56:37.642573 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":665,"threads_started":1}
I20260808 12:56:37.644258 27232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:56:37.645562 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 27232
I20260808 12:56:37.645761 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:56:37.646404 25244 external_mini_cluster.cc:1428] Setting key c2efd795619b5daff5b81002102f90c9
I20260808 12:56:37.647943 27413 heartbeater.cc:347] Connected to a master server at 127.24.167.62:42591
I20260808 12:56:37.648774 27413 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:37.649514 27413 heartbeater.cc:511] Master 127.24.167.62:42591 requested a full tablet report, sending...
I20260808 12:56:37.651039 26763 ts_manager.cc:194] Registered new tserver with Master: 8f037916297f43c3a388aaeec0691b02 (127.24.167.3:37555)
I20260808 12:56:37.651939 26763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:33225
I20260808 12:56:37.662285 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 12:56:37 dist-test-slave-tk9z krb5kdc[26702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260808 12:56:37.770475 25244 mini_kdc.cc:317] Time spent kinit for test-user: real 0.106s user 0.005s sys 0.007s
Aug 08 12:56:37 dist-test-slave-tk9z krb5kdc[26702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:37.789453 27264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.775337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.24.167.62:42591)
0808 12:56:37.775564 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:37.775569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:37.775711 (+ 142us) server_negotiation.cc:407] Connection header received
0808 12:56:37.775883 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:37.775887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:37.775970 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:37.776067 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:37.777592 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.778308 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.779646 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.780056 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.780150 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.784170 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:37.784175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:37.784178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:37.784211 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:37.786902 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:37.787588 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:37.787592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:37.787594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:37.787716 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:37.788097 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:37.788101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:37.788103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:37.788399 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:37.788626 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:37.788868 (+ 242us) server_negotiation.cc:299] Negotiation successful
0808 12:56:37.789040 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 12:56:37.793851 26763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:56:37.801155 26763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:56:37.826673 27430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.814058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:37555 (local address 127.0.0.1:59724)
0808 12:56:37.815047 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:37.815083 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:37.815259 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:37.816464 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:37.816494 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:37.816539 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:37.817069 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.817085 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.820046 (+ 2961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.820055 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.823093 (+ 3038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.823117 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.825586 (+ 2469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.825592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.825829 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.825893 (+ 64us) client_negotiation.cc:769] Sending connection context
0808 12:56:37.826024 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 12:56:37.826234 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":608,"threads_started":1}
I20260808 12:56:37.827533 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.814457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59724 (local address 127.24.167.3:37555)
0808 12:56:37.815479 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:37.815488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:37.815525 (+ 37us) server_negotiation.cc:407] Connection header received
0808 12:56:37.815627 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:37.815636 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:37.815916 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:37.816135 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:37.817306 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.819819 (+ 2513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.823347 (+ 3528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.825243 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.825384 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.827113 (+ 1729us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:37.827171 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 12:56:37.827297 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":650,"threads_started":1}
I20260808 12:56:37.831748 27324 tablet_service.cc:1467] Processing CreateTablet for tablet de8d22ee82e049c78346fbfcfdfa2a36 (DEFAULT_TABLE table=test-table [id=7e9f364e0e2249d29b5fec3c542f2454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:37.833413 27324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de8d22ee82e049c78346fbfcfdfa2a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:37.834780 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.816768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.24.167.1:38983)
0808 12:56:37.826608 (+ 9840us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:37.826620 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:37.826645 (+ 25us) server_negotiation.cc:407] Connection header received
0808 12:56:37.826725 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:37.826730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:37.827929 (+ 1199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:37.828140 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:37.829141 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.830151 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.832469 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.833667 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.833805 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.834016 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:37.834264 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 12:56:37.834461 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9645,"thread_start_us":9466,"threads_started":1}
I20260808 12:56:37.834800 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.816120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:38983 (local address 127.0.0.1:36682)
0808 12:56:37.825552 (+ 9432us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:37.825580 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:37.825735 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:37.828139 (+ 2404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:37.828143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:37.828342 (+ 199us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:37.828941 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.828952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.830342 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.830347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.832316 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.832329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.833881 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.833887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.834054 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.834095 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 12:56:37.834202 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 12:56:37.834331 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9244,"mutex_wait_us":20,"spinlock_wait_cycles":355072,"thread_start_us":8934,"threads_started":1}
I20260808 12:56:37.835987 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.815843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.24.167.2:39019)
0808 12:56:37.827162 (+ 11319us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:37.827170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:37.827202 (+ 32us) server_negotiation.cc:407] Connection header received
0808 12:56:37.827277 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:37.827282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:37.827448 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:37.827597 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:37.829119 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.830660 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.833225 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.834617 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.834763 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.834982 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:37.835529 (+ 547us) server_negotiation.cc:299] Negotiation successful
0808 12:56:37.835689 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11133,"thread_start_us":10184,"threads_started":1}
I20260808 12:56:37.835996 27433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.815509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:39019 (local address 127.0.0.1:42640)
0808 12:56:37.816037 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:37.816056 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:37.816175 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:37.827652 (+ 11477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:37.827656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:37.828334 (+ 678us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:37.828897 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.828916 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.830833 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.830838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.833044 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.833059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.835122 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.835128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.835321 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.835354 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 12:56:37.835471 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 12:56:37.835606 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":3,"spinlock_wait_cycles":896,"thread_start_us":246,"threads_started":1}
I20260808 12:56:37.839079 26941 tablet_service.cc:1467] Processing CreateTablet for tablet de8d22ee82e049c78346fbfcfdfa2a36 (DEFAULT_TABLE table=test-table [id=7e9f364e0e2249d29b5fec3c542f2454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:37.840004 27129 tablet_service.cc:1467] Processing CreateTablet for tablet de8d22ee82e049c78346fbfcfdfa2a36 (DEFAULT_TABLE table=test-table [id=7e9f364e0e2249d29b5fec3c542f2454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:37.840654 26941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de8d22ee82e049c78346fbfcfdfa2a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:37.841341 27129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de8d22ee82e049c78346fbfcfdfa2a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:37.848034 27440 tablet_bootstrap.cc:492] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02: Bootstrap starting.
I20260808 12:56:37.851322 27440 tablet_bootstrap.cc:654] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:37.852720 27440 log.cc:824] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02: Log is configured to *not* fsync() on all Append() calls
I20260808 12:56:37.856673 27440 tablet_bootstrap.cc:492] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02: No bootstrap required, opened a new log
I20260808 12:56:37.856936 27443 tablet_bootstrap.cc:492] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd: Bootstrap starting.
I20260808 12:56:37.857266 27440 ts_tablet_manager.cc:1397] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260808 12:56:37.858727 27444 tablet_bootstrap.cc:492] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7: Bootstrap starting.
I20260808 12:56:37.862568 27443 tablet_bootstrap.cc:654] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:37.862730 27444 tablet_bootstrap.cc:654] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:37.864714 27444 log.cc:824] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7: Log is configured to *not* fsync() on all Append() calls
I20260808 12:56:37.864674 27443 log.cc:824] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd: Log is configured to *not* fsync() on all Append() calls
I20260808 12:56:37.868854 27443 tablet_bootstrap.cc:492] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd: No bootstrap required, opened a new log
I20260808 12:56:37.869200 27443 ts_tablet_manager.cc:1397] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260808 12:56:37.869248 27444 tablet_bootstrap.cc:492] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7: No bootstrap required, opened a new log
I20260808 12:56:37.869654 27444 ts_tablet_manager.cc:1397] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260808 12:56:37.874464 27440 raft_consensus.cc:348] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } }
I20260808 12:56:37.875065 27440 raft_consensus.cc:374] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:37.875242 27440 raft_consensus.cc:729] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f037916297f43c3a388aaeec0691b02, State: Initialized, Role: FOLLOWER
I20260808 12:56:37.876333 27440 consensus_queue.cc:260] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } }
I20260808 12:56:37.882434 27443 raft_consensus.cc:348] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } }
I20260808 12:56:37.883544 27443 raft_consensus.cc:374] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:37.883781 27443 raft_consensus.cc:729] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89a412042609494cb20203ee62d34dfd, State: Initialized, Role: FOLLOWER
I20260808 12:56:37.883829 27413 heartbeater.cc:503] Master 127.24.167.62:42591 was elected leader, sending a full tablet report...
I20260808 12:56:37.884558 27443 consensus_queue.cc:260] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } }
I20260808 12:56:37.886534 27226 heartbeater.cc:503] Master 127.24.167.62:42591 was elected leader, sending a full tablet report...
I20260808 12:56:37.887782 27440 ts_tablet_manager.cc:1428] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02: Time spent starting tablet: real 0.030s user 0.008s sys 0.016s
I20260808 12:56:37.889338 27443 ts_tablet_manager.cc:1428] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd: Time spent starting tablet: real 0.020s user 0.009s sys 0.010s
W20260808 12:56:37.898617 27414 tablet.cc:2369] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:56:37.898885 27444 raft_consensus.cc:348] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } }
I20260808 12:56:37.899535 27444 raft_consensus.cc:374] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:37.899729 27444 raft_consensus.cc:729] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30b4e9353ad4d07b0d47069d25327d7, State: Initialized, Role: FOLLOWER
I20260808 12:56:37.900552 27444 consensus_queue.cc:260] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } }
I20260808 12:56:37.902860 27444 ts_tablet_manager.cc:1428] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7: Time spent starting tablet: real 0.033s user 0.014s sys 0.008s
I20260808 12:56:37.965106 27451 raft_consensus.cc:482] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:56:37.965525 27451 raft_consensus.cc:504] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } }
I20260808 12:56:37.967823 27451 leader_election.cc:288] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f037916297f43c3a388aaeec0691b02 (127.24.167.3:37555), 89a412042609494cb20203ee62d34dfd (127.24.167.2:39019)
I20260808 12:56:37.978395 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.968407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:33645 (local address 127.24.167.3:37555)
0808 12:56:37.968821 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:37.968828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:37.969399 (+ 571us) server_negotiation.cc:407] Connection header received
0808 12:56:37.969723 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:37.969727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:37.969792 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:37.969843 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:37.971592 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.972798 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.975837 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.977054 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.977341 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.977440 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:37.977814 (+ 374us) server_negotiation.cc:299] Negotiation successful
0808 12:56:37.977946 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260808 12:56:37.978646 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.968055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:37555 (local address 127.24.167.1:33645)
0808 12:56:37.969257 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:37.969281 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:37.969491 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:37.970080 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:37.970087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:37.970110 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:37.971429 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.971444 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.972941 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.972950 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.975647 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.975667 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.977356 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.977364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.977617 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.977643 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 12:56:37.977771 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 12:56:37.977920 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"thread_start_us":783,"threads_started":1}
I20260808 12:56:37.980255 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.969372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:56701 (local address 127.24.167.2:39019)
0808 12:56:37.969648 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:37.969657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:37.970738 (+ 1081us) server_negotiation.cc:407] Connection header received
0808 12:56:37.971134 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:37.971143 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:37.971228 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:37.971300 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:37.972466 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.973628 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.976384 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.978579 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.978799 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.978907 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:37.979717 (+ 810us) server_negotiation.cc:299] Negotiation successful
0808 12:56:37.979906 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 12:56:37.980537 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:37.969269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:39019 (local address 127.24.167.1:56701)
0808 12:56:37.970609 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:37.970655 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:37.970935 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:37.971505 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:37.971508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:37.971528 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:37.972276 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.972289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.973824 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.973828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.976163 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:37.976179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:37.979162 (+ 2983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:37.979171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:37.979494 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:37.979550 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 12:56:37.979655 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 12:56:37.979835 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"thread_start_us":696,"threads_started":1}
I20260808 12:56:37.980144 27357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8d22ee82e049c78346fbfcfdfa2a36" candidate_uuid: "e30b4e9353ad4d07b0d47069d25327d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f037916297f43c3a388aaeec0691b02" is_pre_election: true
I20260808 12:56:37.981099 27357 raft_consensus.cc:2454] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30b4e9353ad4d07b0d47069d25327d7 in term 0.
I20260808 12:56:37.982107 26873 leader_election.cc:302] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f037916297f43c3a388aaeec0691b02, e30b4e9353ad4d07b0d47069d25327d7; no voters:
I20260808 12:56:37.982061 27157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8d22ee82e049c78346fbfcfdfa2a36" candidate_uuid: "e30b4e9353ad4d07b0d47069d25327d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89a412042609494cb20203ee62d34dfd" is_pre_election: true
I20260808 12:56:37.982811 27451 raft_consensus.cc:2781] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:56:37.982972 27451 raft_consensus.cc:482] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:56:37.983072 27451 raft_consensus.cc:3037] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:37.983000 27157 raft_consensus.cc:2454] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30b4e9353ad4d07b0d47069d25327d7 in term 0.
W20260808 12:56:38.024868 27032 tablet.cc:2369] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 12:56:38.061563 27229 tablet.cc:2369] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:56:38.076256 27451 raft_consensus.cc:504] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } }
I20260808 12:56:38.077005 27451 leader_election.cc:288] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f037916297f43c3a388aaeec0691b02 (127.24.167.3:37555), 89a412042609494cb20203ee62d34dfd (127.24.167.2:39019)
I20260808 12:56:38.077876 27157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8d22ee82e049c78346fbfcfdfa2a36" candidate_uuid: "e30b4e9353ad4d07b0d47069d25327d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89a412042609494cb20203ee62d34dfd"
I20260808 12:56:38.077973 27357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8d22ee82e049c78346fbfcfdfa2a36" candidate_uuid: "e30b4e9353ad4d07b0d47069d25327d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f037916297f43c3a388aaeec0691b02"
I20260808 12:56:38.078219 27157 raft_consensus.cc:3037] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:38.078346 27357 raft_consensus.cc:3037] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:38.081316 27157 raft_consensus.cc:2454] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30b4e9353ad4d07b0d47069d25327d7 in term 1.
I20260808 12:56:38.081967 26873 leader_election.cc:302] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89a412042609494cb20203ee62d34dfd, e30b4e9353ad4d07b0d47069d25327d7; no voters:
I20260808 12:56:38.082496 27451 raft_consensus.cc:2781] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:38.083706 27451 raft_consensus.cc:686] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [term 1 LEADER]: Becoming Leader. State: Replica: e30b4e9353ad4d07b0d47069d25327d7, State: Running, Role: LEADER
I20260808 12:56:38.084453 27451 consensus_queue.cc:237] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } }
I20260808 12:56:38.090801 26761 catalog_manager.cc:5495] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e30b4e9353ad4d07b0d47069d25327d7 (127.24.167.1). New cstate: current_term: 1 leader_uuid: "e30b4e9353ad4d07b0d47069d25327d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } health_report { overall_health: HEALTHY } } }
I20260808 12:56:38.131455 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:38.124723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.24.167.1:38983)
0808 12:56:38.124908 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:38.124912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:38.124945 (+ 33us) server_negotiation.cc:407] Connection header received
0808 12:56:38.125034 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:38.125038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:38.125166 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:38.125292 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:56:38.125814 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:38.126924 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:38.128764 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:38.129334 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:38.129435 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:38.130048 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:56:38.130674 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:56:38.130781 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:38.131007 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 12:56:38.131228 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 12:56:38.142318 27157 raft_consensus.cc:1261] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd [term 1 FOLLOWER]: Refusing update from remote peer e30b4e9353ad4d07b0d47069d25327d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:56:38.143462 27454 consensus_queue.cc:1035] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:56:38.177872 26763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:56:38.178282 26763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:56:38.179597 27357 raft_consensus.cc:2454] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30b4e9353ad4d07b0d47069d25327d7 in term 1.
I20260808 12:56:38.180886 27356 raft_consensus.cc:1261] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02 [term 1 FOLLOWER]: Refusing update from remote peer e30b4e9353ad4d07b0d47069d25327d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:56:38.183312 27454 consensus_queue.cc:1035] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:56:38.186199 26763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c: Sending DeleteTablet for 3 replicas of tablet de8d22ee82e049c78346fbfcfdfa2a36
I20260808 12:56:38.187930 26941 tablet_service.cc:1514] Processing DeleteTablet for tablet de8d22ee82e049c78346fbfcfdfa2a36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:38 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:36682
I20260808 12:56:38.187911 27129 tablet_service.cc:1514] Processing DeleteTablet for tablet de8d22ee82e049c78346fbfcfdfa2a36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:38 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:42640
I20260808 12:56:38.188273 27324 tablet_service.cc:1514] Processing DeleteTablet for tablet de8d22ee82e049c78346fbfcfdfa2a36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:38 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:59724
I20260808 12:56:38.191039 26763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:56:38.192104 26763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:56:38.197116 27466 tablet_replica.cc:330] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7: stopping tablet replica
I20260808 12:56:38.199368 27467 tablet_replica.cc:330] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02: stopping tablet replica
I20260808 12:56:38.202992 27466 raft_consensus.cc:2229] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:56:38.204128 27466 raft_consensus.cc:2258] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:38.205839 27465 tablet_replica.cc:330] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd: stopping tablet replica
I20260808 12:56:38.208701 27129 tablet_service.cc:1467] Processing CreateTablet for tablet c5d469a5de5b41459ee44bfeeb47ab96 (DEFAULT_TABLE table=test-table [id=68bd09bccaad48b989b2662e2c3aed3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:38.209126 26941 tablet_service.cc:1467] Processing CreateTablet for tablet c5d469a5de5b41459ee44bfeeb47ab96 (DEFAULT_TABLE table=test-table [id=68bd09bccaad48b989b2662e2c3aed3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:38.209545 27129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d469a5de5b41459ee44bfeeb47ab96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:38.209806 26941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d469a5de5b41459ee44bfeeb47ab96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:38.206271 27465 raft_consensus.cc:2229] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:56:38.217042 27324 tablet_service.cc:1467] Processing CreateTablet for tablet c5d469a5de5b41459ee44bfeeb47ab96 (DEFAULT_TABLE table=test-table [id=68bd09bccaad48b989b2662e2c3aed3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:38.217736 27324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d469a5de5b41459ee44bfeeb47ab96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:38.217769 27465 raft_consensus.cc:2258] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:38.219141 27466 ts_tablet_manager.cc:1905] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:38.219553 27443 tablet_bootstrap.cc:492] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd: Bootstrap starting.
I20260808 12:56:38.222193 27465 ts_tablet_manager.cc:1905] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:38.222487 27443 tablet_bootstrap.cc:654] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:38.232975 27443 tablet_bootstrap.cc:492] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd: No bootstrap required, opened a new log
I20260808 12:56:38.233340 27443 ts_tablet_manager.cc:1397] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260808 12:56:38.234748 27443 raft_consensus.cc:348] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.235170 27443 raft_consensus.cc:374] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:38.235471 27443 raft_consensus.cc:729] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89a412042609494cb20203ee62d34dfd, State: Initialized, Role: FOLLOWER
I20260808 12:56:38.235971 27443 consensus_queue.cc:260] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.237063 27440 tablet_bootstrap.cc:492] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02: Bootstrap starting.
I20260808 12:56:38.239253 27444 tablet_bootstrap.cc:492] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7: Bootstrap starting.
I20260808 12:56:38.239378 27440 tablet_bootstrap.cc:654] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:38.239990 27443 ts_tablet_manager.cc:1428] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260808 12:56:38.241497 27444 tablet_bootstrap.cc:654] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:38.244427 27467 raft_consensus.cc:2229] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:56:38.245720 27467 raft_consensus.cc:2258] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:38.252373 27440 tablet_bootstrap.cc:492] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02: No bootstrap required, opened a new log
I20260808 12:56:38.252754 27440 ts_tablet_manager.cc:1397] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20260808 12:56:38.253082 27467 ts_tablet_manager.cc:1905] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:38.255488 27440 raft_consensus.cc:348] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.255939 27440 raft_consensus.cc:374] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:38.256183 27440 raft_consensus.cc:729] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f037916297f43c3a388aaeec0691b02, State: Initialized, Role: FOLLOWER
I20260808 12:56:38.256664 27440 consensus_queue.cc:260] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.257447 27440 ts_tablet_manager.cc:1428] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 12:56:38.257479 27444 tablet_bootstrap.cc:492] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7: No bootstrap required, opened a new log
I20260808 12:56:38.257645 27444 ts_tablet_manager.cc:1397] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.004s
I20260808 12:56:38.260382 27444 raft_consensus.cc:348] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.260684 27444 raft_consensus.cc:374] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:38.260907 27444 raft_consensus.cc:729] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30b4e9353ad4d07b0d47069d25327d7, State: Initialized, Role: FOLLOWER
I20260808 12:56:38.261432 27444 consensus_queue.cc:260] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.262356 27444 ts_tablet_manager.cc:1428] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 12:56:38.291849 27467 ts_tablet_manager.cc:1918] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:56:38.291904 27465 ts_tablet_manager.cc:1918] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:56:38.292088 27465 log.cc:1196] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/de8d22ee82e049c78346fbfcfdfa2a36
I20260808 12:56:38.292142 27467 log.cc:1196] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/de8d22ee82e049c78346fbfcfdfa2a36
I20260808 12:56:38.292806 27467 ts_tablet_manager.cc:1939] T de8d22ee82e049c78346fbfcfdfa2a36 P 8f037916297f43c3a388aaeec0691b02: Deleting consensus metadata
I20260808 12:56:38.292806 27465 ts_tablet_manager.cc:1939] T de8d22ee82e049c78346fbfcfdfa2a36 P 89a412042609494cb20203ee62d34dfd: Deleting consensus metadata
I20260808 12:56:38.292815 27466 ts_tablet_manager.cc:1918] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:56:38.293144 27466 log.cc:1196] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/de8d22ee82e049c78346fbfcfdfa2a36
I20260808 12:56:38.293915 27466 ts_tablet_manager.cc:1939] T de8d22ee82e049c78346fbfcfdfa2a36 P e30b4e9353ad4d07b0d47069d25327d7: Deleting consensus metadata
I20260808 12:56:38.367204 26744 catalog_manager.cc:4841] TS 8f037916297f43c3a388aaeec0691b02 (127.24.167.3:37555): tablet de8d22ee82e049c78346fbfcfdfa2a36 (table test-table [id=7e9f364e0e2249d29b5fec3c542f2454]) successfully deleted
I20260808 12:56:38.376891 26744 catalog_manager.cc:4841] TS 89a412042609494cb20203ee62d34dfd (127.24.167.2:39019): tablet de8d22ee82e049c78346fbfcfdfa2a36 (table test-table [id=7e9f364e0e2249d29b5fec3c542f2454]) successfully deleted
I20260808 12:56:38.379889 26744 catalog_manager.cc:4841] TS e30b4e9353ad4d07b0d47069d25327d7 (127.24.167.1:38983): tablet de8d22ee82e049c78346fbfcfdfa2a36 (table test-table [id=7e9f364e0e2249d29b5fec3c542f2454]) successfully deleted
I20260808 12:56:38.569228 27450 raft_consensus.cc:482] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:56:38.569583 27450 raft_consensus.cc:504] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.571879 27450 leader_election.cc:288] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e30b4e9353ad4d07b0d47069d25327d7 (127.24.167.1:38983), 8f037916297f43c3a388aaeec0691b02 (127.24.167.3:37555)
I20260808 12:56:38.585173 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:38.572417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:55671 (local address 127.24.167.1:38983)
0808 12:56:38.572726 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:38.572732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:38.573296 (+ 564us) server_negotiation.cc:407] Connection header received
0808 12:56:38.573771 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:38.573778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:38.573992 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:38.574083 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:38.575366 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:38.576960 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:38.581177 (+ 4217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:38.583135 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:38.583469 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:38.583576 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:38.584400 (+ 824us) server_negotiation.cc:299] Negotiation successful
0808 12:56:38.584562 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 12:56:38.585601 27479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:38.572118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:38983 (local address 127.24.167.2:55671)
0808 12:56:38.573137 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:38.573163 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:38.573502 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:38.574483 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:38.574487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:38.574519 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:38.575042 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:38.575061 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:38.577188 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:38.577193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:38.580872 (+ 3679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:38.580894 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:38.583815 (+ 2921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:38.583822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:38.584090 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:38.584135 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 12:56:38.584946 (+ 811us) client_negotiation.cc:240] Negotiation successful
0808 12:56:38.585123 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":526,"threads_started":1}
I20260808 12:56:38.587895 26969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d469a5de5b41459ee44bfeeb47ab96" candidate_uuid: "89a412042609494cb20203ee62d34dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30b4e9353ad4d07b0d47069d25327d7" is_pre_election: true
I20260808 12:56:38.588949 26969 raft_consensus.cc:2454] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89a412042609494cb20203ee62d34dfd in term 0.
I20260808 12:56:38.590260 27065 leader_election.cc:302] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89a412042609494cb20203ee62d34dfd, e30b4e9353ad4d07b0d47069d25327d7; no voters:
I20260808 12:56:38.590998 27450 raft_consensus.cc:2781] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:56:38.591173 27450 raft_consensus.cc:482] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:56:38.591285 27450 raft_consensus.cc:3037] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:38.594547 27480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:38.573132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:37555 (local address 127.24.167.2:60699)
0808 12:56:38.584757 (+ 11625us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:38.584774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:38.584937 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:38.585625 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:38.585633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:38.585667 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:38.586567 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:38.586598 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:38.588290 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:38.588298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:38.590941 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:38.590971 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:38.593510 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:38.593518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:38.593794 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:38.593830 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 12:56:38.593933 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 12:56:38.594096 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11500,"thread_start_us":11327,"threads_started":1}
I20260808 12:56:38.594947 27482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:38.573253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:60699 (local address 127.24.167.3:37555)
0808 12:56:38.584428 (+ 11175us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:38.584437 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:38.584882 (+ 445us) server_negotiation.cc:407] Connection header received
0808 12:56:38.585120 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:38.585126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:38.585205 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:38.585302 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:38.586859 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:38.588155 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:38.591180 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:38.593161 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:38.593343 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:38.593428 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:38.594024 (+ 596us) server_negotiation.cc:299] Negotiation successful
0808 12:56:38.594191 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11013,"thread_start_us":10823,"threads_started":1}
I20260808 12:56:38.595705 27356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d469a5de5b41459ee44bfeeb47ab96" candidate_uuid: "89a412042609494cb20203ee62d34dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f037916297f43c3a388aaeec0691b02" is_pre_election: true
I20260808 12:56:38.596135 27356 raft_consensus.cc:2454] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89a412042609494cb20203ee62d34dfd in term 0.
I20260808 12:56:38.599880 27450 raft_consensus.cc:504] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.600883 27450 leader_election.cc:288] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [CANDIDATE]: Term 1 election: Requested vote from peers e30b4e9353ad4d07b0d47069d25327d7 (127.24.167.1:38983), 8f037916297f43c3a388aaeec0691b02 (127.24.167.3:37555)
I20260808 12:56:38.601553 26969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d469a5de5b41459ee44bfeeb47ab96" candidate_uuid: "89a412042609494cb20203ee62d34dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30b4e9353ad4d07b0d47069d25327d7"
I20260808 12:56:38.601854 26969 raft_consensus.cc:3037] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:38.602006 27356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d469a5de5b41459ee44bfeeb47ab96" candidate_uuid: "89a412042609494cb20203ee62d34dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f037916297f43c3a388aaeec0691b02"
I20260808 12:56:38.602335 27356 raft_consensus.cc:3037] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:38.704152 27356 raft_consensus.cc:2454] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89a412042609494cb20203ee62d34dfd in term 1.
I20260808 12:56:38.704232 26969 raft_consensus.cc:2454] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89a412042609494cb20203ee62d34dfd in term 1.
I20260808 12:56:38.704953 27065 leader_election.cc:302] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89a412042609494cb20203ee62d34dfd, 8f037916297f43c3a388aaeec0691b02; no voters:
I20260808 12:56:38.705420 27450 raft_consensus.cc:2781] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:38.706092 27450 raft_consensus.cc:686] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [term 1 LEADER]: Becoming Leader. State: Replica: 89a412042609494cb20203ee62d34dfd, State: Running, Role: LEADER
I20260808 12:56:38.706965 27450 consensus_queue.cc:237] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.711912 26763 catalog_manager.cc:5495] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 89a412042609494cb20203ee62d34dfd (127.24.167.2). New cstate: current_term: 1 leader_uuid: "89a412042609494cb20203ee62d34dfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } health_report { overall_health: UNKNOWN } } }
Aug 08 12:56:38 dist-test-slave-tk9z krb5kdc[26702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:38.833655 27496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:38.813510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42591 (local address 127.0.0.1:47768)
0808 12:56:38.814788 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:38.814805 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:38.814991 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:38.816056 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:38.816060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:38.817145 (+ 1085us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:38.817693 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:38.817714 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:38.819140 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:38.819148 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:38.820374 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:38.820387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:38.821398 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:38.821404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:38.821641 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:38.822545 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:38.822563 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:38.825705 (+ 3142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:38.830189 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:38.830201 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:38.830207 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:38.831002 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:38.831451 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:38.831455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:38.831457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:38.831688 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:38.832394 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:38.832400 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:38.832575 (+ 175us) client_negotiation.cc:769] Sending connection context
0808 12:56:38.832770 (+ 195us) client_negotiation.cc:240] Negotiation successful
0808 12:56:38.833002 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"thread_start_us":927,"threads_started":1}
I20260808 12:56:38.833700 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:38.813478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47768 (local address 127.24.167.62:42591)
0808 12:56:38.814547 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:38.814565 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:38.814902 (+ 337us) server_negotiation.cc:407] Connection header received
0808 12:56:38.815842 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:38.815852 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:38.815912 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:38.816031 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:38.817855 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:38.819009 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:38.820551 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:38.821225 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:38.821377 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:38.825951 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:38.825957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:38.825960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:38.825990 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:38.830053 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:38.831109 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:38.831113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:38.831115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:38.831324 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:38.831832 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:38.831836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:38.831838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:38.832227 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:38.832386 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:38.832940 (+ 554us) server_negotiation.cc:299] Negotiation successful
0808 12:56:38.833121 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":568,"threads_started":1}
I20260808 12:56:38.839592 26763 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:47768:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 12:56:38.840898 26763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:56:38.848706 27129 tablet_service.cc:1467] Processing CreateTablet for tablet 41b745cd7931477d874c8d2e0c4dc69b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f66169153814242934f5453e7287b22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 12:56:38.848647 26941 tablet_service.cc:1467] Processing CreateTablet for tablet 41b745cd7931477d874c8d2e0c4dc69b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f66169153814242934f5453e7287b22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 12:56:38.849361 26941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b745cd7931477d874c8d2e0c4dc69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:38.849452 27129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b745cd7931477d874c8d2e0c4dc69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:38.851455 27324 tablet_service.cc:1467] Processing CreateTablet for tablet 41b745cd7931477d874c8d2e0c4dc69b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f66169153814242934f5453e7287b22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 12:56:38.852097 27324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b745cd7931477d874c8d2e0c4dc69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:38.859277 27497 tablet_bootstrap.cc:492] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd: Bootstrap starting.
I20260808 12:56:38.860737 27501 tablet_bootstrap.cc:492] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7: Bootstrap starting.
I20260808 12:56:38.862968 27501 tablet_bootstrap.cc:654] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:38.864177 27497 tablet_bootstrap.cc:654] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:38.866225 27501 tablet_bootstrap.cc:492] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7: No bootstrap required, opened a new log
I20260808 12:56:38.866626 27501 ts_tablet_manager.cc:1397] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260808 12:56:38.867121 27498 tablet_bootstrap.cc:492] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02: Bootstrap starting.
I20260808 12:56:38.867843 27501 raft_consensus.cc:348] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.868376 27501 raft_consensus.cc:374] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:38.868602 27501 raft_consensus.cc:729] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30b4e9353ad4d07b0d47069d25327d7, State: Initialized, Role: FOLLOWER
I20260808 12:56:38.868913 27501 consensus_queue.cc:260] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.870100 27501 ts_tablet_manager.cc:1428] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 12:56:38.870779 27498 tablet_bootstrap.cc:654] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:38.871107 27504 tablet_replica.cc:439] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } }
I20260808 12:56:38.871622 27504 tablet_replica.cc:442] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:56:38.874177 27497 tablet_bootstrap.cc:492] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd: No bootstrap required, opened a new log
I20260808 12:56:38.874425 27497 ts_tablet_manager.cc:1397] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260808 12:56:38.874557 27498 tablet_bootstrap.cc:492] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02: No bootstrap required, opened a new log
I20260808 12:56:38.874717 27498 ts_tablet_manager.cc:1397] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 12:56:38.875330 27497 raft_consensus.cc:348] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.875629 27497 raft_consensus.cc:374] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:38.875576 27498 raft_consensus.cc:348] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.875784 27497 raft_consensus.cc:729] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89a412042609494cb20203ee62d34dfd, State: Initialized, Role: FOLLOWER
I20260808 12:56:38.875905 27498 raft_consensus.cc:374] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:38.875983 27497 consensus_queue.cc:260] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.876083 27498 raft_consensus.cc:729] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f037916297f43c3a388aaeec0691b02, State: Initialized, Role: FOLLOWER
I20260808 12:56:38.876386 27498 consensus_queue.cc:260] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.876459 27450 tablet_replica.cc:439] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } }
I20260808 12:56:38.876670 27497 ts_tablet_manager.cc:1428] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260808 12:56:38.876821 27450 tablet_replica.cc:442] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:56:38.878012 27498 ts_tablet_manager.cc:1428] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 12:56:38.877853 27506 tablet_replica.cc:439] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } }
I20260808 12:56:38.878321 27506 tablet_replica.cc:442] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:56:38.915145 27450 raft_consensus.cc:482] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:56:38.916122 27450 raft_consensus.cc:504] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:38.916914 27450 leader_election.cc:288] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e30b4e9353ad4d07b0d47069d25327d7 (127.24.167.1:38983), 8f037916297f43c3a388aaeec0691b02 (127.24.167.3:37555)
I20260808 12:56:38.917824 26969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b745cd7931477d874c8d2e0c4dc69b" candidate_uuid: "89a412042609494cb20203ee62d34dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30b4e9353ad4d07b0d47069d25327d7" is_pre_election: true
I20260808 12:56:38.918195 26969 raft_consensus.cc:2454] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89a412042609494cb20203ee62d34dfd in term 0.
I20260808 12:56:38.918749 27065 leader_election.cc:302] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89a412042609494cb20203ee62d34dfd, e30b4e9353ad4d07b0d47069d25327d7; no voters:
I20260808 12:56:38.918864 27356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b745cd7931477d874c8d2e0c4dc69b" candidate_uuid: "89a412042609494cb20203ee62d34dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f037916297f43c3a388aaeec0691b02" is_pre_election: true
I20260808 12:56:38.919178 27356 raft_consensus.cc:2454] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89a412042609494cb20203ee62d34dfd in term 0.
I20260808 12:56:38.919679 27450 raft_consensus.cc:2781] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:56:38.919798 27450 raft_consensus.cc:482] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:56:38.920029 27450 raft_consensus.cc:3037] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:39.011267 27450 raft_consensus.cc:504] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:39.012233 27450 leader_election.cc:288] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [CANDIDATE]: Term 1 election: Requested vote from peers e30b4e9353ad4d07b0d47069d25327d7 (127.24.167.1:38983), 8f037916297f43c3a388aaeec0691b02 (127.24.167.3:37555)
I20260808 12:56:39.012820 26969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b745cd7931477d874c8d2e0c4dc69b" candidate_uuid: "89a412042609494cb20203ee62d34dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30b4e9353ad4d07b0d47069d25327d7"
I20260808 12:56:39.013185 26969 raft_consensus.cc:3037] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:39.013063 27356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b745cd7931477d874c8d2e0c4dc69b" candidate_uuid: "89a412042609494cb20203ee62d34dfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f037916297f43c3a388aaeec0691b02"
I20260808 12:56:39.013365 27356 raft_consensus.cc:3037] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:39.015857 26969 raft_consensus.cc:2454] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89a412042609494cb20203ee62d34dfd in term 1.
I20260808 12:56:39.016494 27065 leader_election.cc:302] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89a412042609494cb20203ee62d34dfd, e30b4e9353ad4d07b0d47069d25327d7; no voters:
I20260808 12:56:39.016858 27450 raft_consensus.cc:2781] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:39.017046 27450 raft_consensus.cc:686] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [term 1 LEADER]: Becoming Leader. State: Replica: 89a412042609494cb20203ee62d34dfd, State: Running, Role: LEADER
I20260808 12:56:39.017426 27450 consensus_queue.cc:237] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } }
I20260808 12:56:39.019562 27483 tablet_replica.cc:439] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd: TxnStatusTablet state changed. Reason: New leader 89a412042609494cb20203ee62d34dfd. Latest consensus state: current_term: 1 leader_uuid: "89a412042609494cb20203ee62d34dfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } }
I20260808 12:56:39.020180 27483 tablet_replica.cc:442] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd: This TxnStatusTablet replica's current role is: LEADER
I20260808 12:56:39.020916 26763 catalog_manager.cc:5495] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 89a412042609494cb20203ee62d34dfd (127.24.167.2). New cstate: current_term: 1 leader_uuid: "89a412042609494cb20203ee62d34dfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } health_report { overall_health: UNKNOWN } } }
I20260808 12:56:39.021694 27509 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 12:56:39.022078 27509 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 12:56:39.070779 27496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:39.060012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:39019 (local address 127.0.0.1:42652)
0808 12:56:39.060293 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:39.060315 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:39.060477 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:39.062081 (+ 1604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:39.062087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:39.062102 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 12:56:39.062339 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:39.062354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.064775 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.064781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:39.066716 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:39.066734 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.067951 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.067958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:39.068166 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:39.068305 (+ 139us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 12:56:39.069931 (+ 1626us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 12:56:39.069958 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 12:56:39.070021 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 12:56:39.070221 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 12:56:39.073071 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:39.060060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.24.167.2:39019)
0808 12:56:39.061494 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:39.061501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:39.061529 (+ 28us) server_negotiation.cc:407] Connection header received
0808 12:56:39.061683 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:39.061690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:39.061812 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:39.062027 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:56:39.062732 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.064596 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.066955 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.067749 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.068503 (+ 754us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:39.068770 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:56:39.069756 (+ 986us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:56:39.071375 (+ 1619us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:39.071511 (+ 136us) server_negotiation.cc:299] Negotiation successful
0808 12:56:39.071597 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1189,"thread_start_us":968,"threads_started":1}
I20260808 12:56:39.081423 26969 raft_consensus.cc:1261] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7 [term 1 FOLLOWER]: Refusing update from remote peer 89a412042609494cb20203ee62d34dfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:56:39.082463 27483 consensus_queue.cc:1035] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:56:39.094754 27504 tablet_replica.cc:439] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7: TxnStatusTablet state changed. Reason: New leader 89a412042609494cb20203ee62d34dfd. Latest consensus state: current_term: 1 leader_uuid: "89a412042609494cb20203ee62d34dfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } }
I20260808 12:56:39.095080 27504 tablet_replica.cc:442] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:56:39.107576 27483 tablet_replica.cc:439] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89a412042609494cb20203ee62d34dfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } }
I20260808 12:56:39.108306 27483 tablet_replica.cc:442] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd: This TxnStatusTablet replica's current role is: LEADER
I20260808 12:56:39.112598 27519 consensus_queue.cc:1035] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:56:39.114696 27356 raft_consensus.cc:2454] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89a412042609494cb20203ee62d34dfd in term 1.
I20260808 12:56:39.115290 27357 raft_consensus.cc:1261] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02 [term 1 FOLLOWER]: Refusing update from remote peer 89a412042609494cb20203ee62d34dfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:56:39.117064 27519 consensus_queue.cc:1035] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:56:39.123487 27525 consensus_queue.cc:1035] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:56:39.132544 27504 tablet_replica.cc:439] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89a412042609494cb20203ee62d34dfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } }
I20260808 12:56:39.132874 27504 tablet_replica.cc:442] T 41b745cd7931477d874c8d2e0c4dc69b P e30b4e9353ad4d07b0d47069d25327d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:56:39.145605 27506 tablet_replica.cc:439] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02: TxnStatusTablet state changed. Reason: New leader 89a412042609494cb20203ee62d34dfd. Latest consensus state: current_term: 1 leader_uuid: "89a412042609494cb20203ee62d34dfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } }
I20260808 12:56:39.146055 27506 tablet_replica.cc:442] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:56:39.147768 27528 tablet_replica.cc:439] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89a412042609494cb20203ee62d34dfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } }
I20260808 12:56:39.148270 27528 tablet_replica.cc:442] T 41b745cd7931477d874c8d2e0c4dc69b P 8f037916297f43c3a388aaeec0691b02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:56:39.151183 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:39.130360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.24.167.2:39019)
0808 12:56:39.130587 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:39.130594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:39.143197 (+ 12603us) server_negotiation.cc:407] Connection header received
0808 12:56:39.143482 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:39.143490 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:39.143600 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:39.143709 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:56:39.144528 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.145601 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.149107 (+ 3506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.149737 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.149891 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:39.150436 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:56:39.150548 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:56:39.150651 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:39.150813 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 12:56:39.150900 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 12:56:39.175138 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:39.166153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:39019 (local address 127.24.167.2:51763)
0808 12:56:39.167026 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:39.167044 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:39.167211 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:39.167732 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:39.167735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:39.167744 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 12:56:39.168273 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:39.168284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.170149 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.170153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:39.172068 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:39.172087 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.173512 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.173518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:39.173761 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:39.173830 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 12:56:39.174321 (+ 491us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 12:56:39.174333 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 12:56:39.174483 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 12:56:39.174594 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":38,"thread_start_us":530,"threads_started":1}
I20260808 12:56:39.175146 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:39.166194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:51763 (local address 127.24.167.2:39019)
0808 12:56:39.166510 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:39.166526 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:39.167127 (+ 601us) server_negotiation.cc:407] Connection header received
0808 12:56:39.167405 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:39.167412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:39.167602 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:39.167708 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:56:39.168576 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.169999 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.172292 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.173239 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.173393 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:39.174051 (+ 658us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:56:39.174182 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:56:39.174300 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:39.174516 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 12:56:39.174601 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":54}
I20260808 12:56:39.178797 27527 tablet_replica.cc:439] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89a412042609494cb20203ee62d34dfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30b4e9353ad4d07b0d47069d25327d7" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 38983 } } peers { permanent_uuid: "89a412042609494cb20203ee62d34dfd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39019 } } peers { permanent_uuid: "8f037916297f43c3a388aaeec0691b02" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 37555 } } }
I20260808 12:56:39.179418 27527 tablet_replica.cc:442] T 41b745cd7931477d874c8d2e0c4dc69b P 89a412042609494cb20203ee62d34dfd: This TxnStatusTablet replica's current role is: LEADER
I20260808 12:56:39.255779 26763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:56:39.256100 26763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:56:39.259841 26763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3fa10f97aad34da198e63292d9dbb90c: Sending DeleteTablet for 3 replicas of tablet c5d469a5de5b41459ee44bfeeb47ab96
I20260808 12:56:39.260951 26941 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d469a5de5b41459ee44bfeeb47ab96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:39 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:36682
I20260808 12:56:39.261022 27324 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d469a5de5b41459ee44bfeeb47ab96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:39 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:59724
I20260808 12:56:39.262105 27538 tablet_replica.cc:330] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02: stopping tablet replica
I20260808 12:56:39.261127 27128 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d469a5de5b41459ee44bfeeb47ab96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:39 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:42640
I20260808 12:56:39.262449 27539 tablet_replica.cc:330] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7: stopping tablet replica
I20260808 12:56:39.262558 27538 raft_consensus.cc:2229] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:56:39.262714 27539 raft_consensus.cc:2229] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:56:39.263257 27539 raft_consensus.cc:2258] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:39.263284 27538 raft_consensus.cc:2258] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:39.264523 27539 ts_tablet_manager.cc:1905] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:39.264515 27538 ts_tablet_manager.cc:1905] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:39.271445 27541 tablet_replica.cc:330] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd: stopping tablet replica
I20260808 12:56:39.271741 27541 raft_consensus.cc:2229] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:56:39.272924 27541 raft_consensus.cc:2258] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:39.275674 27541 ts_tablet_manager.cc:1905] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:39.293569 27539 ts_tablet_manager.cc:1918] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 12:56:39.293625 27538 ts_tablet_manager.cc:1918] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 12:56:39.293802 27539 log.cc:1196] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/c5d469a5de5b41459ee44bfeeb47ab96
I20260808 12:56:39.293879 27538 log.cc:1196] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/c5d469a5de5b41459ee44bfeeb47ab96
I20260808 12:56:39.294448 27539 ts_tablet_manager.cc:1939] T c5d469a5de5b41459ee44bfeeb47ab96 P e30b4e9353ad4d07b0d47069d25327d7: Deleting consensus metadata
I20260808 12:56:39.294821 27538 ts_tablet_manager.cc:1939] T c5d469a5de5b41459ee44bfeeb47ab96 P 8f037916297f43c3a388aaeec0691b02: Deleting consensus metadata
I20260808 12:56:39.296031 26744 catalog_manager.cc:4841] TS e30b4e9353ad4d07b0d47069d25327d7 (127.24.167.1:38983): tablet c5d469a5de5b41459ee44bfeeb47ab96 (table test-table [id=68bd09bccaad48b989b2662e2c3aed3e]) successfully deleted
I20260808 12:56:39.296190 27541 ts_tablet_manager.cc:1918] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 12:56:39.296332 27541 log.cc:1196] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/c5d469a5de5b41459ee44bfeeb47ab96
I20260808 12:56:39.296406 26744 catalog_manager.cc:4841] TS 8f037916297f43c3a388aaeec0691b02 (127.24.167.3:37555): tablet c5d469a5de5b41459ee44bfeeb47ab96 (table test-table [id=68bd09bccaad48b989b2662e2c3aed3e]) successfully deleted
Aug 08 12:56:39 dist-test-slave-tk9z krb5kdc[26702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:56:39.297120 27541 ts_tablet_manager.cc:1939] T c5d469a5de5b41459ee44bfeeb47ab96 P 89a412042609494cb20203ee62d34dfd: Deleting consensus metadata
I20260808 12:56:39.298566 26744 catalog_manager.cc:4841] TS 89a412042609494cb20203ee62d34dfd (127.24.167.2:39019): tablet c5d469a5de5b41459ee44bfeeb47ab96 (table test-table [id=68bd09bccaad48b989b2662e2c3aed3e]) successfully deleted
I20260808 12:56:39.305619 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:39.276901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.24.167.1:38983)
0808 12:56:39.278078 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:39.278084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:39.284343 (+ 6259us) server_negotiation.cc:407] Connection header received
0808 12:56:39.284626 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:39.284630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:39.284697 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:39.284862 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:39.287334 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.288389 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.289964 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.290680 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.290824 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:39.297850 (+ 7026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:39.297864 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:39.297880 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:39.298001 (+ 121us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:39.302121 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:39.303066 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:39.303074 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:39.303076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:39.303257 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:39.303734 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:39.303738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:39.303740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:39.304152 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:39.304516 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:39.304812 (+ 296us) server_negotiation.cc:299] Negotiation successful
0808 12:56:39.305136 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1011,"thread_start_us":827,"threads_started":1}
W20260808 12:56:39.306593 27022 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36700
I20260808 12:56:39.328202 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:39.310205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.24.167.62:42591)
0808 12:56:39.310551 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:39.310558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:39.311052 (+ 494us) server_negotiation.cc:407] Connection header received
0808 12:56:39.311204 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:39.311210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:39.311306 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:39.311418 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:39.312933 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.314230 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.315583 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.316671 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.316799 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:39.319808 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:39.319815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:39.319820 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:39.319868 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:39.324414 (+ 4546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:39.325308 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:39.325314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:39.325317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:39.325630 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:39.326174 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:39.326179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:39.326182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:39.326747 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:39.327094 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:39.327557 (+ 463us) server_negotiation.cc:299] Negotiation successful
0808 12:56:39.327846 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
W20260808 12:56:39.329018 26763 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47772
I20260808 12:56:39.332868 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26851
I20260808 12:56:39.353147 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:39.334654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36704 (local address 127.24.167.1:38983)
0808 12:56:39.334944 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:39.334955 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:39.338944 (+ 3989us) server_negotiation.cc:407] Connection header received
0808 12:56:39.339196 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:39.339204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:39.339293 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:39.339418 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:39.340844 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.341630 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.343343 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.343837 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.343994 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:39.347229 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:39.347234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:39.347238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:39.347277 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:39.350246 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:39.350987 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:39.350993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:39.350996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:39.351145 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:39.351597 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:39.351604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:39.351612 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:39.351886 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:39.352155 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:39.352474 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 12:56:39.352709 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260808 12:56:39.353803 27022 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36704
W20260808 12:56:39.354537 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26851: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 12:56:39.354841 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 26851
I20260808 12:56:39.393834 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 27035
Aug 08 12:56:39 dist-test-slave-tk9z krb5kdc[26702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 12:56:39.412354 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:39.394754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.24.167.2:39019)
0808 12:56:39.395037 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:39.395042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:39.395277 (+ 235us) server_negotiation.cc:407] Connection header received
0808 12:56:39.395514 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:39.395521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:39.395626 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:39.395756 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:39.397081 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.397928 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.399450 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.399977 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.400129 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:39.405665 (+ 5536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:39.405687 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:39.405703 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:39.405745 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:39.409058 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:39.409825 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:39.409839 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:39.409841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:39.410020 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:39.410501 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:39.410508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:39.410511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:39.410944 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:39.411346 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:39.411682 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 12:56:39.411988 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260808 12:56:39.413115 27220 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42668
W20260808 12:56:39.414018 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 27035: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 12:56:39.414268 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 27035
I20260808 12:56:39.442900 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 27232
Aug 08 12:56:39 dist-test-slave-tk9z krb5kdc[26702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 12:56:39.458747 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:39.443794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.24.167.3:37555)
0808 12:56:39.444413 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:39.444418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:39.444437 (+ 19us) server_negotiation.cc:407] Connection header received
0808 12:56:39.444498 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:39.444502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:39.444566 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:39.444675 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:39.445935 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.446996 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.448237 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:39.448641 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:39.448738 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:39.452900 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:39.452918 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:39.452936 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:39.452980 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:39.456100 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:39.456779 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:39.456787 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:39.456789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:39.456934 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:39.457259 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:39.457262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:39.457264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:39.457657 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:39.457876 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:39.458138 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 12:56:39.458315 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":352,"threads_started":1}
W20260808 12:56:39.459551 27408 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59740
W20260808 12:56:39.460196 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 27232: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 12:56:39.460386 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 27232
I20260808 12:56:39.486680 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26715
I20260808 12:56:39.487875 26817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:56:39.608177 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 26715
2026-08-08T12:56:39Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5703 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:56:39 dist-test-slave-tk9z krb5kdc[27554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:56:39 dist-test-slave-tk9z krb5kdc[27554](info): set up 2 sockets
Aug 08 12:56:39 dist-test-slave-tk9z krb5kdc[27554](info): commencing operation
krb5kdc: starting...
W20260808 12:56:40.900348 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:56:41 dist-test-slave-tk9z krb5kdc[27554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:56:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:56:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 12:56:41.107973 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:39171
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:39171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:41.437855 27574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:41.438150 27574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:41.438244 27574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:41.445407 27574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:56:41.445511 27574 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 12:56:41.445554 27574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:41.445618 27574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:41.445669 27574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:56:41.445725 27574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:56:41.455348 27574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:39171
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:39171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:41.456557 27574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:41.457839 27574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:41.469827 27581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:41.469898 27583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:41.469872 27579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:41.470767 27574 server_base.cc:1034] running on GCE node
I20260808 12:56:41.471470 27574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:41.472764 27574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:41.474001 27574 hybrid_clock.cc:648] HybridClock initialized: now 1786193801473932 us; error 38 us; skew 500 ppm
I20260808 12:56:41.474360 27574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:41 dist-test-slave-tk9z krb5kdc[27554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193801, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:41.481343 27574 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:56:41.482939 27574 webserver.cc:460] Webserver started at http://127.24.167.62:39203/ using document root <none> and password file <none>
I20260808 12:56:41.483668 27574 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:41.483783 27574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:41.484146 27574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:41.487324 27574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "600a14dc4ddb4cabb162562fd78841f3"
format_stamp: "Formatted at 2026-08-08 12:56:41 on dist-test-slave-tk9z"
server_key: "18a66cc28b9959e13e37708bbec2a109"
server_key_iv: "7c9a15ab2e33037a450c03f0f1bb5fb3"
server_key_version: "encryptionkey@0"
I20260808 12:56:41.488126 27574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "600a14dc4ddb4cabb162562fd78841f3"
format_stamp: "Formatted at 2026-08-08 12:56:41 on dist-test-slave-tk9z"
server_key: "18a66cc28b9959e13e37708bbec2a109"
server_key_iv: "7c9a15ab2e33037a450c03f0f1bb5fb3"
server_key_version: "encryptionkey@0"
I20260808 12:56:41.493309 27574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 12:56:41.496750 27594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:41.498064 27574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 12:56:41.498240 27574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "600a14dc4ddb4cabb162562fd78841f3"
format_stamp: "Formatted at 2026-08-08 12:56:41 on dist-test-slave-tk9z"
server_key: "18a66cc28b9959e13e37708bbec2a109"
server_key_iv: "7c9a15ab2e33037a450c03f0f1bb5fb3"
server_key_version: "encryptionkey@0"
I20260808 12:56:41.498538 27574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:41.531178 27574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:41.535164 27574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:41.535431 27574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:41.558259 27574 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:39171
I20260808 12:56:41.558279 27677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:39171 every 8 connection(s)
I20260808 12:56:41.559752 27574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:56:41.563629 27679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:41.567302 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 27574
I20260808 12:56:41.567559 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:56:41.568410 25244 external_mini_cluster.cc:1428] Setting key 328c46e8a1b373cb141d5aa194e88b23
I20260808 12:56:41.577512 27679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3: Bootstrap starting.
I20260808 12:56:41.581389 27679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:41.582787 27679 log.cc:824] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3: Log is configured to *not* fsync() on all Append() calls
Aug 08 12:56:41 dist-test-slave-tk9z krb5kdc[27554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:41.585867 27679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3: No bootstrap required, opened a new log
I20260808 12:56:41.592334 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:41.570699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.24.167.62:39171)
0808 12:56:41.571540 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:41.571552 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:41.571596 (+ 44us) server_negotiation.cc:407] Connection header received
0808 12:56:41.573256 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:41.573280 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:41.573699 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:41.574036 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:41.575951 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:41.577232 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:41.578972 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:41.579680 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:41.579809 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:41.584900 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:41.584907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:41.584921 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:41.584958 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:41.588163 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:41.588975 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:41.588982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:41.588988 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:41.589178 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:41.589635 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:41.589639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:41.589640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:41.590156 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:41.590455 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:41.590918 (+ 463us) server_negotiation.cc:299] Negotiation successful
0808 12:56:41.591317 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":404,"threads_started":1}
I20260808 12:56:41.595363 27679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a14dc4ddb4cabb162562fd78841f3" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39171 } }
I20260808 12:56:41.595781 27679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:41.595889 27679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600a14dc4ddb4cabb162562fd78841f3, State: Initialized, Role: FOLLOWER
I20260808 12:56:41.596458 27679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a14dc4ddb4cabb162562fd78841f3" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39171 } }
I20260808 12:56:41.596720 27679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:56:41.596874 27679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:56:41.597039 27679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:41.599452 27679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a14dc4ddb4cabb162562fd78841f3" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39171 } }
I20260808 12:56:41.599958 27679 leader_election.cc:302] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 600a14dc4ddb4cabb162562fd78841f3; no voters:
I20260808 12:56:41.600528 27679 leader_election.cc:288] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:56:41.600677 27690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:41.601437 27690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [term 1 LEADER]: Becoming Leader. State: Replica: 600a14dc4ddb4cabb162562fd78841f3, State: Running, Role: LEADER
I20260808 12:56:41.602074 27690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a14dc4ddb4cabb162562fd78841f3" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39171 } }
I20260808 12:56:41.602924 27679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:56:41.605280 27692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 600a14dc4ddb4cabb162562fd78841f3. Latest consensus state: current_term: 1 leader_uuid: "600a14dc4ddb4cabb162562fd78841f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a14dc4ddb4cabb162562fd78841f3" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39171 } } }
I20260808 12:56:41.605756 27692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [sys.catalog]: This master's current role is: LEADER
I20260808 12:56:41.606390 27691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "600a14dc4ddb4cabb162562fd78841f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a14dc4ddb4cabb162562fd78841f3" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39171 } } }
I20260808 12:56:41.606696 27691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3 [sys.catalog]: This master's current role is: LEADER
I20260808 12:56:41.609385 27701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:56:41.615127 27701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:56:41.624058 27701 catalog_manager.cc:1347] Generated new cluster ID: 41c558ffc9224c669b3566328bf99d2b
I20260808 12:56:41.624193 27701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:56:41.647692 27701 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:56:41.649258 27701 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:56:41.662281 27701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3: Generated new TSK 0
I20260808 12:56:41.663292 27701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 12:56:41.722726 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:39171
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:42.070783 27718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:42.071110 27718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:42.071301 27718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:42.078568 27718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:42.078744 27718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:42.078913 27718 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 12:56:42.079005 27718 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 12:56:42.079125 27718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:56:42.089119 27718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.167.62:39171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:42.090601 27718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:42.092025 27718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:42.105436 27724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:42.105485 27725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:42.105968 27727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:42.106297 27718 server_base.cc:1034] running on GCE node
I20260808 12:56:42.107079 27718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:42.108073 27718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:42.109288 27718 hybrid_clock.cc:648] HybridClock initialized: now 1786193802109228 us; error 57 us; skew 500 ppm
I20260808 12:56:42.109639 27718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:42 dist-test-slave-tk9z krb5kdc[27554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193802, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:42.113348 27718 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 12:56:42.114846 27718 webserver.cc:460] Webserver started at http://127.24.167.1:44655/ using document root <none> and password file <none>
I20260808 12:56:42.115542 27718 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:42.115685 27718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:42.115968 27718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:42.118957 27718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "77541b5634d7460c9a6147298588cd0a"
format_stamp: "Formatted at 2026-08-08 12:56:42 on dist-test-slave-tk9z"
server_key: "56ac8c280225cc6cecb62c9b8dcd0b6d"
server_key_iv: "a19f0936819bce755d37d4d1ca4b372d"
server_key_version: "encryptionkey@0"
I20260808 12:56:42.119697 27718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "77541b5634d7460c9a6147298588cd0a"
format_stamp: "Formatted at 2026-08-08 12:56:42 on dist-test-slave-tk9z"
server_key: "56ac8c280225cc6cecb62c9b8dcd0b6d"
server_key_iv: "a19f0936819bce755d37d4d1ca4b372d"
server_key_version: "encryptionkey@0"
I20260808 12:56:42.124931 27718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 12:56:42.128415 27738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:42.129843 27718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 12:56:42.130023 27718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "77541b5634d7460c9a6147298588cd0a"
format_stamp: "Formatted at 2026-08-08 12:56:42 on dist-test-slave-tk9z"
server_key: "56ac8c280225cc6cecb62c9b8dcd0b6d"
server_key_iv: "a19f0936819bce755d37d4d1ca4b372d"
server_key_version: "encryptionkey@0"
I20260808 12:56:42.130234 27718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:42.144591 27718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:42.148890 27718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:42.149212 27718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:42.152103 27718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:42.152279 27718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:42.152426 27718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:42.152529 27718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 12:56:42 dist-test-slave-tk9z krb5kdc[27554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193802, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:42.181794 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:42.152979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:37783 (local address 127.24.167.62:39171)
0808 12:56:42.153520 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:42.153528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:42.154831 (+ 1303us) server_negotiation.cc:407] Connection header received
0808 12:56:42.157488 (+ 2657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:42.157494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:42.157604 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:42.157734 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:42.160822 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.162165 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.165450 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.166211 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.166399 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:42.172730 (+ 6331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:42.172742 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:42.172745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:42.172779 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:42.176149 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:42.177319 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:42.177329 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:42.177336 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:42.177548 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:42.178212 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:42.178221 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:42.178227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:42.178710 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:42.178891 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:42.180897 (+ 2006us) server_negotiation.cc:299] Negotiation successful
0808 12:56:42.181224 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":274,"threads_started":1}
I20260808 12:56:42.184381 27748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:42.153686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:39171 (local address 127.24.167.1:37783)
0808 12:56:42.154654 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:42.154699 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:42.157146 (+ 2447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:42.158091 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:42.158107 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:42.159276 (+ 1169us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:42.160548 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:42.160597 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.162355 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.162362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:42.165247 (+ 2885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:42.165288 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.166535 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.166543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:42.166950 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:42.167951 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:42.167996 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:42.172417 (+ 4421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:42.176357 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:42.176371 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:42.176392 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:42.177143 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:42.177712 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:42.177716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:42.177723 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:42.178010 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:42.178990 (+ 980us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:42.178998 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:42.179659 (+ 661us) client_negotiation.cc:769] Sending connection context
0808 12:56:42.180057 (+ 398us) client_negotiation.cc:240] Negotiation successful
0808 12:56:42.182218 (+ 2161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":414,"threads_started":1}
I20260808 12:56:42.198616 27718 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:42153
I20260808 12:56:42.198676 27895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:42153 every 8 connection(s)
I20260808 12:56:42.200024 27718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:56:42.202991 27896 heartbeater.cc:347] Connected to a master server at 127.24.167.62:39171
I20260808 12:56:42.203308 27896 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:42.204156 27896 heartbeater.cc:511] Master 127.24.167.62:39171 requested a full tablet report, sending...
I20260808 12:56:42.206698 27616 ts_manager.cc:194] Registered new tserver with Master: 77541b5634d7460c9a6147298588cd0a (127.24.167.1:42153)
I20260808 12:56:42.207580 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 27718
I20260808 12:56:42.207813 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:56:42.208266 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:42.192382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:49031 (local address 127.24.167.62:39171)
0808 12:56:42.192594 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:42.192602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:42.192848 (+ 246us) server_negotiation.cc:407] Connection header received
0808 12:56:42.193168 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:42.193172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:42.193240 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:42.193299 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:42.195234 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.196489 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.198616 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.199183 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.199330 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:42.201970 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:42.201975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:42.201978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:42.202035 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:42.204883 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:42.205824 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:42.205830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:42.205832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:42.206031 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:42.206420 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:42.206425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:42.206427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:42.206819 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:42.206989 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:42.207539 (+ 550us) server_negotiation.cc:299] Negotiation successful
0808 12:56:42.207835 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":76}
I20260808 12:56:42.208307 27871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:42.191850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:39171 (local address 127.24.167.1:49031)
0808 12:56:42.192719 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:42.192748 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:42.192954 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:42.193617 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:42.193622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:42.194443 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:42.194973 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:42.194993 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.196643 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.196649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:42.198364 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:42.198435 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.199420 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.199430 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:42.199624 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:42.200368 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:42.200389 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:42.201759 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:42.205206 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:42.205212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:42.205216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:42.205702 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:42.206154 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:42.206158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:42.206160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:42.206286 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:42.207031 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:42.207035 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:42.207198 (+ 163us) client_negotiation.cc:769] Sending connection context
0808 12:56:42.207425 (+ 227us) client_negotiation.cc:240] Negotiation successful
0808 12:56:42.207784 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":453,"threads_started":1}
I20260808 12:56:42.208598 25244 external_mini_cluster.cc:1428] Setting key 7c86a602280fe646c69c06b1a7e72147
I20260808 12:56:42.208904 27616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:37783
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 12:56:42.255081 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:39171
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:42.589458 27902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:42.589805 27902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:42.589990 27902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:42.597383 27902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:42.597544 27902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:42.597704 27902 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 12:56:42.597796 27902 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 12:56:42.597939 27902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:56:42.608151 27902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.167.62:39171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:42.609562 27902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:42.611234 27902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:42.624771 27914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:42.624771 27909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:42.625026 27908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:42.625839 27902 server_base.cc:1034] running on GCE node
I20260808 12:56:42.626526 27902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:42.627489 27902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:42.628738 27902 hybrid_clock.cc:648] HybridClock initialized: now 1786193802628703 us; error 53 us; skew 500 ppm
I20260808 12:56:42.629065 27902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:42 dist-test-slave-tk9z krb5kdc[27554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193802, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:42.632342 27902 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 12:56:42.633953 27902 webserver.cc:460] Webserver started at http://127.24.167.2:46671/ using document root <none> and password file <none>
I20260808 12:56:42.634728 27902 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:42.634886 27902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:42.635221 27902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:42.638226 27902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "599699167f5f402db480c097187ff662"
format_stamp: "Formatted at 2026-08-08 12:56:42 on dist-test-slave-tk9z"
server_key: "eb4641edf00fd6ae473b782e89204383"
server_key_iv: "09202343c6a45bdbcde38a94f07e8b87"
server_key_version: "encryptionkey@0"
I20260808 12:56:42.639061 27902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "599699167f5f402db480c097187ff662"
format_stamp: "Formatted at 2026-08-08 12:56:42 on dist-test-slave-tk9z"
server_key: "eb4641edf00fd6ae473b782e89204383"
server_key_iv: "09202343c6a45bdbcde38a94f07e8b87"
server_key_version: "encryptionkey@0"
I20260808 12:56:42.644397 27902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 12:56:42.647969 27922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:42.649286 27902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 12:56:42.649482 27902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "599699167f5f402db480c097187ff662"
format_stamp: "Formatted at 2026-08-08 12:56:42 on dist-test-slave-tk9z"
server_key: "eb4641edf00fd6ae473b782e89204383"
server_key_iv: "09202343c6a45bdbcde38a94f07e8b87"
server_key_version: "encryptionkey@0"
I20260808 12:56:42.649722 27902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:42.666353 27902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:42.670926 27902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:42.671201 27902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:42.674131 27902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:42.674243 27902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:42.674433 27902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:42.674538 27902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 12:56:42 dist-test-slave-tk9z krb5kdc[27554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193802, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:42.701138 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:42.674836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:53179 (local address 127.24.167.62:39171)
0808 12:56:42.675069 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:42.675075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:42.676913 (+ 1838us) server_negotiation.cc:407] Connection header received
0808 12:56:42.680220 (+ 3307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:42.680230 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:42.680331 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:42.680480 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:42.683230 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.684313 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.687184 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.687978 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.688124 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:42.692292 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:42.692297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:42.692300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:42.692335 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:42.695972 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:42.697342 (+ 1370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:42.697348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:42.697352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:42.697548 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:42.698219 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:42.698226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:42.698229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:42.698608 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:42.698826 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:42.700687 (+ 1861us) server_negotiation.cc:299] Negotiation successful
0808 12:56:42.700871 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 12:56:42.702883 27939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:42.675471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:39171 (local address 127.24.167.2:53179)
0808 12:56:42.676689 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:42.676741 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:42.679755 (+ 3014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:42.680788 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:42.680804 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:42.681874 (+ 1070us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:42.683002 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:42.683035 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.684571 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.684575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:42.686958 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:42.686981 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.688256 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.688264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:42.688484 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:42.689301 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:42.689325 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:42.692037 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:42.696198 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:42.696217 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:42.696245 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:42.697168 (+ 923us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:42.697695 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:42.697701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:42.697707 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:42.698047 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:42.698791 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:42.698805 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:42.699486 (+ 681us) client_negotiation.cc:769] Sending connection context
0808 12:56:42.699916 (+ 430us) client_negotiation.cc:240] Negotiation successful
0808 12:56:42.700722 (+ 806us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"thread_start_us":565,"threads_started":1}
I20260808 12:56:42.736665 27902 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:44625
I20260808 12:56:42.736778 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:42.721606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:50323 (local address 127.24.167.62:39171)
0808 12:56:42.721787 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:42.721791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:42.722532 (+ 741us) server_negotiation.cc:407] Connection header received
0808 12:56:42.722750 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:42.722754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:42.722819 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:42.722931 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:42.723943 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.724745 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.726070 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.726559 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.726697 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:42.728886 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:42.728897 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:42.728905 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:42.728965 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:42.732700 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:42.733793 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:42.733801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:42.733807 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:42.734021 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:42.734606 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:42.734611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:42.734614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:42.735105 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:42.735295 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:42.736125 (+ 830us) server_negotiation.cc:299] Negotiation successful
0808 12:56:42.736399 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 12:56:42.736711 28056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:42.721718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:39171 (local address 127.24.167.2:50323)
0808 12:56:42.722344 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:42.722361 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:42.722584 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:42.722973 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:42.722978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:42.723486 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:42.723785 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:42.723794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.724877 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.724883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:42.725915 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:42.725927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:42.726802 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:42.726809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:42.726992 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:42.727705 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:42.727723 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:42.728644 (+ 921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:42.732844 (+ 4200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:42.732849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:42.732853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:42.733648 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:42.734202 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:42.734206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:42.734208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:42.734467 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:42.735223 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:42.735227 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:42.735511 (+ 284us) client_negotiation.cc:769] Sending connection context
0808 12:56:42.735695 (+ 184us) client_negotiation.cc:240] Negotiation successful
0808 12:56:42.736063 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":347,"threads_started":1}
I20260808 12:56:42.736760 28096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:44625 every 8 connection(s)
I20260808 12:56:42.738207 27902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:56:42.740474 28097 heartbeater.cc:347] Connected to a master server at 127.24.167.62:39171
I20260808 12:56:42.740763 28097 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:42.741461 28097 heartbeater.cc:511] Master 127.24.167.62:39171 requested a full tablet report, sending...
I20260808 12:56:42.742877 27615 ts_manager.cc:194] Registered new tserver with Master: 599699167f5f402db480c097187ff662 (127.24.167.2:44625)
I20260808 12:56:42.743779 27615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:53179
I20260808 12:56:42.745242 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 27902
I20260808 12:56:42.745457 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:56:42.745898 25244 external_mini_cluster.cc:1428] Setting key c16c6bc7da25fc846d115204a30a69a9
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 12:56:42.790010 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:39171
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:43.130759 28105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:43.131120 28105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:43.131296 28105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:43.138870 28105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:43.139029 28105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:43.139192 28105 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 12:56:43.139284 28105 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 12:56:43.139433 28105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:56:43.149513 28105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.167.62:39171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:43.150923 28105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:43.152287 28105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:43.166133 28113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:43.166394 28114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:43.166476 28116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:43.167187 28105 server_base.cc:1034] running on GCE node
I20260808 12:56:43.167873 28105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:43.169020 28105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:43.170262 28105 hybrid_clock.cc:648] HybridClock initialized: now 1786193803170213 us; error 60 us; skew 500 ppm
I20260808 12:56:43.170629 28105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:43 dist-test-slave-tk9z krb5kdc[27554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:43.173900 28105 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 12:56:43.175500 28105 webserver.cc:460] Webserver started at http://127.24.167.3:38411/ using document root <none> and password file <none>
I20260808 12:56:43.176209 28105 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:43.176366 28105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:43.176707 28105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:43.180058 28105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "e2daa90edaa84c1c915fec66dce4317a"
format_stamp: "Formatted at 2026-08-08 12:56:43 on dist-test-slave-tk9z"
server_key: "a35e40ee085df29aa7e5db426cf192de"
server_key_iv: "24da415dd9c98a1e29a1d5766b45432a"
server_key_version: "encryptionkey@0"
I20260808 12:56:43.180960 28105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "e2daa90edaa84c1c915fec66dce4317a"
format_stamp: "Formatted at 2026-08-08 12:56:43 on dist-test-slave-tk9z"
server_key: "a35e40ee085df29aa7e5db426cf192de"
server_key_iv: "24da415dd9c98a1e29a1d5766b45432a"
server_key_version: "encryptionkey@0"
I20260808 12:56:43.186481 28105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260808 12:56:43.189869 28126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:43.191237 28105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:56:43.191417 28105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "e2daa90edaa84c1c915fec66dce4317a"
format_stamp: "Formatted at 2026-08-08 12:56:43 on dist-test-slave-tk9z"
server_key: "a35e40ee085df29aa7e5db426cf192de"
server_key_iv: "24da415dd9c98a1e29a1d5766b45432a"
server_key_version: "encryptionkey@0"
I20260808 12:56:43.191594 28105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:43.212975 28105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:43.213215 27896 heartbeater.cc:503] Master 127.24.167.62:39171 was elected leader, sending a full tablet report...
I20260808 12:56:43.217851 28105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:43.218170 28105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:43.221202 28105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:43.221361 28105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:43.221498 28105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:43.221604 28105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 12:56:43 dist-test-slave-tk9z krb5kdc[27554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:43.246964 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.221981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:49165 (local address 127.24.167.62:39171)
0808 12:56:43.222172 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:43.222179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:43.223817 (+ 1638us) server_negotiation.cc:407] Connection header received
0808 12:56:43.226893 (+ 3076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:43.226902 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:43.226992 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:43.227082 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:43.229616 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.230532 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.233264 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.233940 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.234080 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.238481 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:43.238486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:43.238490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:43.238524 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:43.242829 (+ 4305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:43.244006 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:43.244011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:43.244013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:43.244148 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:43.244913 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:43.244920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:43.244924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:43.245322 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:43.245502 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:43.246298 (+ 796us) server_negotiation.cc:299] Negotiation successful
0808 12:56:43.246578 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 12:56:43.248075 28137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.222619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:39171 (local address 127.24.167.3:49165)
0808 12:56:43.223628 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:43.223675 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:43.226532 (+ 2857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:43.227423 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:43.227438 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:43.228347 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:43.229336 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.229366 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.230710 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.230717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.233044 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.233064 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.234217 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.234227 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.234565 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.235350 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:43.235378 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:43.238216 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:43.243026 (+ 4810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:43.243044 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:43.243070 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:43.243800 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:43.244304 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:43.244313 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:43.244320 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:43.244737 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:43.245467 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:43.245475 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:43.245886 (+ 411us) client_negotiation.cc:769] Sending connection context
0808 12:56:43.246183 (+ 297us) client_negotiation.cc:240] Negotiation successful
0808 12:56:43.246805 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":408,"threads_started":1}
I20260808 12:56:43.270502 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.254244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:33697 (local address 127.24.167.62:39171)
0808 12:56:43.254492 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:43.254500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:43.255151 (+ 651us) server_negotiation.cc:407] Connection header received
0808 12:56:43.255430 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:43.255437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:43.255523 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:43.255598 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:43.257256 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.258674 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.260611 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.261581 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.261754 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.264170 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:43.264178 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:43.264183 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:43.264232 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:43.267701 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:43.268395 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:43.268403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:43.268409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:43.268553 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:43.268936 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:43.268941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:43.268945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:43.269242 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:43.269411 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:43.269811 (+ 400us) server_negotiation.cc:299] Negotiation successful
0808 12:56:43.270049 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 12:56:43.270601 28204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.254252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:39171 (local address 127.24.167.3:33697)
0808 12:56:43.255045 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:43.255062 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:43.255236 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:43.255775 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:43.255778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:43.256454 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:43.257033 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.257051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.258888 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.258896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.260444 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.260459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.261781 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.261786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.261936 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.262629 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:43.262648 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:43.263967 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:43.267837 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:43.267843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:43.267848 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:43.268262 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:43.268669 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:43.268674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:43.268678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:43.268818 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:43.269364 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:43.269369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:43.269566 (+ 197us) client_negotiation.cc:769] Sending connection context
0808 12:56:43.269716 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 12:56:43.270076 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":452,"threads_started":1}
I20260808 12:56:43.286952 28105 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:34183
I20260808 12:56:43.287045 28303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:34183 every 8 connection(s)
I20260808 12:56:43.288350 28105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:56:43.290716 28304 heartbeater.cc:347] Connected to a master server at 127.24.167.62:39171
I20260808 12:56:43.291023 28304 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:43.290990 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 28105
I20260808 12:56:43.291215 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:56:43.291810 25244 external_mini_cluster.cc:1428] Setting key 89746ac42277d8b08dcff16846dbb8f4
I20260808 12:56:43.291888 28304 heartbeater.cc:511] Master 127.24.167.62:39171 requested a full tablet report, sending...
I20260808 12:56:43.293555 27615 ts_manager.cc:194] Registered new tserver with Master: e2daa90edaa84c1c915fec66dce4317a (127.24.167.3:34183)
I20260808 12:56:43.294615 27615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:49165
I20260808 12:56:43.295320 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 12:56:43 dist-test-slave-tk9z krb5kdc[27554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 12:56:43 dist-test-slave-tk9z krb5kdc[27554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:43.348100 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.334183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.24.167.62:39171)
0808 12:56:43.334462 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:43.334469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:43.334713 (+ 244us) server_negotiation.cc:407] Connection header received
0808 12:56:43.334940 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:43.334945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:43.335017 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:43.335099 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:43.336487 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.337221 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.338489 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.338894 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.339000 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.343276 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:43.343286 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:43.343293 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:43.343336 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:43.345776 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:43.346482 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:43.346485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:43.346488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:43.346610 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:43.346949 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:43.346952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:43.346953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:43.347227 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:43.347418 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:43.347661 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 12:56:43.347842 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 12:56:43.352495 27615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:56:43.359606 27615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:56:43.397246 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.376000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.24.167.1:42153)
0808 12:56:43.387239 (+ 11239us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:43.387251 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:43.388382 (+ 1131us) server_negotiation.cc:407] Connection header received
0808 12:56:43.388492 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:43.388498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:43.388706 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:43.388888 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:43.390070 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.391461 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.394407 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.395909 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.396039 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.396237 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:43.396716 (+ 479us) server_negotiation.cc:299] Negotiation successful
0808 12:56:43.396916 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11060,"thread_start_us":10907,"threads_started":1}
I20260808 12:56:43.397253 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.372659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:34183 (local address 127.0.0.1:57632)
0808 12:56:43.383996 (+ 11337us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:43.384046 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:43.384230 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:43.389322 (+ 5092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:43.389328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:43.389359 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:43.390076 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.390089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.392192 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.392197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.393935 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.393952 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.396082 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.396093 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.396321 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.396391 (+ 70us) client_negotiation.cc:769] Sending connection context
0808 12:56:43.396525 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 12:56:43.396762 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11071,"mutex_wait_us":43,"thread_start_us":15201,"threads_started":1}
I20260808 12:56:43.397965 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.373159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.24.167.2:44625)
0808 12:56:43.386643 (+ 13484us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:43.386652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:43.387923 (+ 1271us) server_negotiation.cc:407] Connection header received
0808 12:56:43.388285 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:43.388298 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:43.388587 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:43.388802 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:43.390233 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.391840 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.394527 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.395591 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.396422 (+ 831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.396645 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:43.397181 (+ 536us) server_negotiation.cc:299] Negotiation successful
0808 12:56:43.397669 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13018,"thread_start_us":13198,"threads_started":1}
I20260808 12:56:43.398015 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.375257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:42153 (local address 127.0.0.1:43986)
0808 12:56:43.388141 (+ 12884us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:43.388159 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:43.388265 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:43.388964 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:43.388972 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:43.389002 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:43.389780 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.389803 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.391616 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.391621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.394186 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.394204 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.396305 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.396312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.396463 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.396493 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 12:56:43.396583 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 12:56:43.397603 (+ 1020us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12746,"mutex_wait_us":862,"thread_start_us":12191,"threads_started":1}
I20260808 12:56:43.398075 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.373084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.24.167.3:34183)
0808 12:56:43.388700 (+ 15616us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:43.388706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:43.388731 (+ 25us) server_negotiation.cc:407] Connection header received
0808 12:56:43.388808 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:43.388813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:43.389093 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:43.389309 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:43.390347 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.392039 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.394117 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.395681 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.395826 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.396116 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:43.397536 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0808 12:56:43.397707 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15445,"thread_start_us":15243,"threads_started":1}
I20260808 12:56:43.398561 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.372746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:44625 (local address 127.0.0.1:46566)
0808 12:56:43.387769 (+ 15023us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:43.387793 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:43.388064 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:43.389140 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:43.389144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:43.389168 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:43.389986 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.390007 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.391989 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.391994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.394349 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.394360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.396784 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.396791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.396960 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.396994 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 12:56:43.397099 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 12:56:43.397249 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14876,"mutex_wait_us":55,"thread_start_us":10833,"threads_started":1}
I20260808 12:56:43.401479 27991 tablet_service.cc:1467] Processing CreateTablet for tablet f7beb321ec614ac3b542d11634486b4a (DEFAULT_TABLE table=test-table [id=cdbb3da180134bc9967eaba58fb70367]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:43.401546 27802 tablet_service.cc:1467] Processing CreateTablet for tablet f7beb321ec614ac3b542d11634486b4a (DEFAULT_TABLE table=test-table [id=cdbb3da180134bc9967eaba58fb70367]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:43.402973 27991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7beb321ec614ac3b542d11634486b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:43.403071 27802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7beb321ec614ac3b542d11634486b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:43.404246 28211 tablet_service.cc:1467] Processing CreateTablet for tablet f7beb321ec614ac3b542d11634486b4a (DEFAULT_TABLE table=test-table [id=cdbb3da180134bc9967eaba58fb70367]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:43.405665 28211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7beb321ec614ac3b542d11634486b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:43.415246 28328 tablet_bootstrap.cc:492] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662: Bootstrap starting.
I20260808 12:56:43.418087 28329 tablet_bootstrap.cc:492] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a: Bootstrap starting.
I20260808 12:56:43.420291 28330 tablet_bootstrap.cc:492] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a: Bootstrap starting.
I20260808 12:56:43.420949 28328 tablet_bootstrap.cc:654] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:43.421627 28329 tablet_bootstrap.cc:654] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:43.422489 28328 log.cc:824] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662: Log is configured to *not* fsync() on all Append() calls
I20260808 12:56:43.422960 28329 log.cc:824] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a: Log is configured to *not* fsync() on all Append() calls
I20260808 12:56:43.423444 28330 tablet_bootstrap.cc:654] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:43.424701 28330 log.cc:824] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a: Log is configured to *not* fsync() on all Append() calls
I20260808 12:56:43.425863 28328 tablet_bootstrap.cc:492] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662: No bootstrap required, opened a new log
I20260808 12:56:43.425964 28329 tablet_bootstrap.cc:492] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a: No bootstrap required, opened a new log
I20260808 12:56:43.426172 28328 ts_tablet_manager.cc:1397] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260808 12:56:43.426244 28329 ts_tablet_manager.cc:1397] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 12:56:43.427896 28330 tablet_bootstrap.cc:492] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a: No bootstrap required, opened a new log
I20260808 12:56:43.428287 28330 ts_tablet_manager.cc:1397] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 12:56:43.434561 28328 raft_consensus.cc:348] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } }
I20260808 12:56:43.435084 28328 raft_consensus.cc:374] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:43.435194 28328 raft_consensus.cc:729] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599699167f5f402db480c097187ff662, State: Initialized, Role: FOLLOWER
I20260808 12:56:43.435781 28328 consensus_queue.cc:260] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } }
I20260808 12:56:43.436932 28097 heartbeater.cc:503] Master 127.24.167.62:39171 was elected leader, sending a full tablet report...
I20260808 12:56:43.437232 28329 raft_consensus.cc:348] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } }
I20260808 12:56:43.437642 28328 ts_tablet_manager.cc:1428] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260808 12:56:43.437669 28329 raft_consensus.cc:374] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:43.437767 28329 raft_consensus.cc:729] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77541b5634d7460c9a6147298588cd0a, State: Initialized, Role: FOLLOWER
I20260808 12:56:43.438339 28329 consensus_queue.cc:260] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } }
I20260808 12:56:43.439539 28330 raft_consensus.cc:348] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } }
I20260808 12:56:43.440836 28330 raft_consensus.cc:374] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:43.440974 28330 raft_consensus.cc:729] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2daa90edaa84c1c915fec66dce4317a, State: Initialized, Role: FOLLOWER
I20260808 12:56:43.441831 28330 consensus_queue.cc:260] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } }
I20260808 12:56:43.444380 28329 ts_tablet_manager.cc:1428] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260808 12:56:43.448194 28304 heartbeater.cc:503] Master 127.24.167.62:39171 was elected leader, sending a full tablet report...
I20260808 12:56:43.448695 28330 ts_tablet_manager.cc:1428] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a: Time spent starting tablet: real 0.020s user 0.008s sys 0.010s
W20260808 12:56:43.452884 27899 tablet.cc:2369] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 12:56:43.490537 28099 tablet.cc:2369] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 12:56:43.540143 28305 tablet.cc:2369] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:56:43.737564 28335 raft_consensus.cc:482] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:56:43.737926 28335 raft_consensus.cc:504] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } }
I20260808 12:56:43.739818 28335 leader_election.cc:288] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2daa90edaa84c1c915fec66dce4317a (127.24.167.3:34183), 599699167f5f402db480c097187ff662 (127.24.167.2:44625)
I20260808 12:56:43.748646 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.740741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:49899 (local address 127.24.167.3:34183)
0808 12:56:43.740999 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:43.741008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:43.741829 (+ 821us) server_negotiation.cc:407] Connection header received
0808 12:56:43.742120 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:43.742125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:43.742218 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:43.742325 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:43.742953 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.743909 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.746074 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.747319 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.747497 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.747601 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:43.748188 (+ 587us) server_negotiation.cc:299] Negotiation successful
0808 12:56:43.748364 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 12:56:43.749379 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.740100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:34183 (local address 127.24.167.1:49899)
0808 12:56:43.741675 (+ 1575us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:43.741703 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:43.741957 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:43.742348 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:43.742353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:43.742392 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:43.742782 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.742794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.744084 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.744089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.745912 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.745925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.747751 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.747760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.747975 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.748025 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 12:56:43.748855 (+ 830us) client_negotiation.cc:240] Negotiation successful
0808 12:56:43.749017 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1388,"thread_start_us":1505,"threads_started":1}
I20260808 12:56:43.749557 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.740433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:39003 (local address 127.24.167.2:44625)
0808 12:56:43.740686 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:43.740696 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:43.742117 (+ 1421us) server_negotiation.cc:407] Connection header received
0808 12:56:43.742242 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:43.742250 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:43.742331 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:43.742445 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:43.743180 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.744270 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.746765 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.748020 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.748150 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.748219 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:43.749157 (+ 938us) server_negotiation.cc:299] Negotiation successful
0808 12:56:43.749291 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 12:56:43.750219 28338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.740269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:44625 (local address 127.24.167.1:39003)
0808 12:56:43.741941 (+ 1672us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:43.741959 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:43.742106 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:43.742558 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:43.742565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:43.742598 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:43.743007 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.743020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.744449 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.744454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.746589 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.746605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.748398 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.748403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.748947 (+ 544us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.748988 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 12:56:43.749742 (+ 754us) client_negotiation.cc:240] Negotiation successful
0808 12:56:43.749910 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1556,"thread_start_us":979,"threads_started":1}
I20260808 12:56:43.750638 28241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7beb321ec614ac3b542d11634486b4a" candidate_uuid: "77541b5634d7460c9a6147298588cd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2daa90edaa84c1c915fec66dce4317a" is_pre_election: true
I20260808 12:56:43.751231 28241 raft_consensus.cc:2454] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77541b5634d7460c9a6147298588cd0a in term 0.
I20260808 12:56:43.751219 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7beb321ec614ac3b542d11634486b4a" candidate_uuid: "77541b5634d7460c9a6147298588cd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599699167f5f402db480c097187ff662" is_pre_election: true
I20260808 12:56:43.751811 28024 raft_consensus.cc:2454] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77541b5634d7460c9a6147298588cd0a in term 0.
I20260808 12:56:43.751886 27742 leader_election.cc:302] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77541b5634d7460c9a6147298588cd0a, e2daa90edaa84c1c915fec66dce4317a; no voters:
I20260808 12:56:43.752378 28335 raft_consensus.cc:2781] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:56:43.752537 28335 raft_consensus.cc:482] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:56:43.752660 28335 raft_consensus.cc:3037] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:43.755318 28335 raft_consensus.cc:504] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } }
I20260808 12:56:43.756052 28335 leader_election.cc:288] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [CANDIDATE]: Term 1 election: Requested vote from peers e2daa90edaa84c1c915fec66dce4317a (127.24.167.3:34183), 599699167f5f402db480c097187ff662 (127.24.167.2:44625)
I20260808 12:56:43.756613 28241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7beb321ec614ac3b542d11634486b4a" candidate_uuid: "77541b5634d7460c9a6147298588cd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2daa90edaa84c1c915fec66dce4317a"
I20260808 12:56:43.756664 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7beb321ec614ac3b542d11634486b4a" candidate_uuid: "77541b5634d7460c9a6147298588cd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599699167f5f402db480c097187ff662"
I20260808 12:56:43.756877 28024 raft_consensus.cc:3037] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:43.756872 28241 raft_consensus.cc:3037] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:43.759229 28024 raft_consensus.cc:2454] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77541b5634d7460c9a6147298588cd0a in term 1.
I20260808 12:56:43.759234 28241 raft_consensus.cc:2454] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77541b5634d7460c9a6147298588cd0a in term 1.
I20260808 12:56:43.759720 27740 leader_election.cc:302] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 599699167f5f402db480c097187ff662, 77541b5634d7460c9a6147298588cd0a; no voters:
I20260808 12:56:43.760114 28335 raft_consensus.cc:2781] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:43.760602 28335 raft_consensus.cc:686] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [term 1 LEADER]: Becoming Leader. State: Replica: 77541b5634d7460c9a6147298588cd0a, State: Running, Role: LEADER
I20260808 12:56:43.761351 28335 consensus_queue.cc:237] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } }
I20260808 12:56:43.766249 27612 catalog_manager.cc:5495] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 77541b5634d7460c9a6147298588cd0a (127.24.167.1). New cstate: current_term: 1 leader_uuid: "77541b5634d7460c9a6147298588cd0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } health_report { overall_health: HEALTHY } } }
Aug 08 12:56:43 dist-test-slave-tk9z krb5kdc[27554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193801, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:43.878042 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.864679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:39171 (local address 127.0.0.1:51410)
0808 12:56:43.865524 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:43.865542 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:43.865669 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:43.865990 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:43.865993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:43.866750 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:43.867004 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.867013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.868122 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.868125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.869078 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:43.869088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.869919 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.869923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:43.870051 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.870713 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:43.870729 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:43.872886 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:43.875903 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:43.875913 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:43.875918 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:43.876344 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:43.876682 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:43.876685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:43.876687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:43.876820 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:43.877276 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:43.877282 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:43.877420 (+ 138us) client_negotiation.cc:769] Sending connection context
0808 12:56:43.877561 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 12:56:43.877753 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":470,"threads_started":1}
I20260808 12:56:43.878199 28359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:43.864710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.24.167.62:39171)
0808 12:56:43.865601 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:43.865605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:43.865627 (+ 22us) server_negotiation.cc:407] Connection header received
0808 12:56:43.865781 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:43.865786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:43.865849 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:43.865942 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:43.867162 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.868007 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.869232 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:43.869747 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:43.869851 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:43.873099 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:43.873104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:43.873107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:43.873134 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:43.875787 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:43.876443 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:43.876460 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:43.876462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:43.876587 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:43.876910 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:43.876913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:43.876915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:43.877160 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:43.877269 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:43.877648 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 12:56:43.877895 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":775,"thread_start_us":508,"threads_started":1}
I20260808 12:56:43.883059 27612 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:51410:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 12:56:43.884183 27612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:56:43.891762 27802 tablet_service.cc:1467] Processing CreateTablet for tablet 349252894a7f40c29a4275787581c0ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aec9625e061d40fda9a5c86938f4de81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 12:56:43.892695 27802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349252894a7f40c29a4275787581c0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:43.892112 27991 tablet_service.cc:1467] Processing CreateTablet for tablet 349252894a7f40c29a4275787581c0ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aec9625e061d40fda9a5c86938f4de81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 12:56:43.892225 28211 tablet_service.cc:1467] Processing CreateTablet for tablet 349252894a7f40c29a4275787581c0ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aec9625e061d40fda9a5c86938f4de81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 12:56:43.893085 27991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349252894a7f40c29a4275787581c0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:43.893095 28211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349252894a7f40c29a4275787581c0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:43.898981 28329 tablet_bootstrap.cc:492] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a: Bootstrap starting.
I20260808 12:56:43.900636 28329 tablet_bootstrap.cc:654] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:43.901139 28328 tablet_bootstrap.cc:492] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662: Bootstrap starting.
I20260808 12:56:43.901141 28330 tablet_bootstrap.cc:492] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a: Bootstrap starting.
I20260808 12:56:43.903103 28328 tablet_bootstrap.cc:654] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:43.903095 28330 tablet_bootstrap.cc:654] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:43.913096 28330 tablet_bootstrap.cc:492] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a: No bootstrap required, opened a new log
I20260808 12:56:43.913319 28330 ts_tablet_manager.cc:1397] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260808 12:56:43.914301 28330 raft_consensus.cc:348] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:43.914629 28330 raft_consensus.cc:374] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:43.914734 28330 raft_consensus.cc:729] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2daa90edaa84c1c915fec66dce4317a, State: Initialized, Role: FOLLOWER
I20260808 12:56:43.914764 28328 tablet_bootstrap.cc:492] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662: No bootstrap required, opened a new log
I20260808 12:56:43.915038 28328 ts_tablet_manager.cc:1397] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260808 12:56:43.915004 28330 consensus_queue.cc:260] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:43.915303 28329 tablet_bootstrap.cc:492] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a: No bootstrap required, opened a new log
I20260808 12:56:43.915475 28329 ts_tablet_manager.cc:1397] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.012s
I20260808 12:56:43.915657 28330 ts_tablet_manager.cc:1428] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 12:56:43.916374 28329 raft_consensus.cc:348] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:43.917840 28329 raft_consensus.cc:374] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:43.917989 28329 raft_consensus.cc:729] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77541b5634d7460c9a6147298588cd0a, State: Initialized, Role: FOLLOWER
I20260808 12:56:43.918318 28329 consensus_queue.cc:260] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:43.916565 28328 raft_consensus.cc:348] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:43.919044 28329 ts_tablet_manager.cc:1428] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 12:56:43.918974 28335 tablet_replica.cc:439] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } }
I20260808 12:56:43.919196 28328 raft_consensus.cc:374] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:43.919076 28337 tablet_replica.cc:439] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } }
I20260808 12:56:43.919353 28335 tablet_replica.cc:442] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:56:43.919399 28337 tablet_replica.cc:442] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:56:43.919402 28328 raft_consensus.cc:729] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599699167f5f402db480c097187ff662, State: Initialized, Role: FOLLOWER
I20260808 12:56:43.919729 28328 consensus_queue.cc:260] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:43.920419 28328 ts_tablet_manager.cc:1428] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260808 12:56:43.920360 28334 tablet_replica.cc:439] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } }
I20260808 12:56:43.920763 28334 tablet_replica.cc:442] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:56:44.024539 28337 raft_consensus.cc:482] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:56:44.024812 28337 raft_consensus.cc:504] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:44.026718 28337 leader_election.cc:288] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77541b5634d7460c9a6147298588cd0a (127.24.167.1:42153), 599699167f5f402db480c097187ff662 (127.24.167.2:44625)
I20260808 12:56:44.033671 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:44.027072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:38859 (local address 127.24.167.1:42153)
0808 12:56:44.028225 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:44.028233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:44.028255 (+ 22us) server_negotiation.cc:407] Connection header received
0808 12:56:44.028337 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:44.028341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:44.028399 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:44.028532 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:44.029099 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.029846 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.031834 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.032660 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.032840 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:44.032918 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:44.033173 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 12:56:44.033312 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1026,"thread_start_us":913,"threads_started":1}
I20260808 12:56:44.033990 28366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:44.026936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:42153 (local address 127.24.167.3:38859)
0808 12:56:44.028019 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:44.028041 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:44.028178 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:44.028549 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:44.028554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:44.028585 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:44.028941 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:44.028950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.029986 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.029993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:44.031681 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:44.031694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.032826 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.032833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:44.033008 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:44.033033 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 12:56:44.033134 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 12:56:44.033265 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":43,"thread_start_us":1315,"threads_started":1}
I20260808 12:56:44.034861 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:44.027332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:49817 (local address 127.24.167.2:44625)
0808 12:56:44.027578 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:44.027587 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:44.028717 (+ 1130us) server_negotiation.cc:407] Connection header received
0808 12:56:44.028972 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:44.028977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:44.029042 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:44.029155 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:56:44.029736 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.030653 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.032558 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.033556 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.033747 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:44.033849 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:44.034356 (+ 507us) server_negotiation.cc:299] Negotiation successful
0808 12:56:44.034525 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 12:56:44.034935 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:44.027141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:44625 (local address 127.24.167.3:49817)
0808 12:56:44.028580 (+ 1439us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:44.028597 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:44.028778 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:44.029176 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:44.029181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:44.029207 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:56:44.029555 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:44.029564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.030797 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.030804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:44.032386 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:44.032399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.033930 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.033938 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:44.034119 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:44.034172 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 12:56:44.034315 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 12:56:44.034503 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"mutex_wait_us":3,"thread_start_us":567,"threads_started":1}
I20260808 12:56:44.035033 27828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349252894a7f40c29a4275787581c0ad" candidate_uuid: "e2daa90edaa84c1c915fec66dce4317a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77541b5634d7460c9a6147298588cd0a" is_pre_election: true
I20260808 12:56:44.035595 27828 raft_consensus.cc:2454] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2daa90edaa84c1c915fec66dce4317a in term 0.
I20260808 12:56:44.035661 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349252894a7f40c29a4275787581c0ad" candidate_uuid: "e2daa90edaa84c1c915fec66dce4317a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599699167f5f402db480c097187ff662" is_pre_election: true
I20260808 12:56:44.035912 28024 raft_consensus.cc:2454] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2daa90edaa84c1c915fec66dce4317a in term 0.
I20260808 12:56:44.036214 28127 leader_election.cc:302] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77541b5634d7460c9a6147298588cd0a, e2daa90edaa84c1c915fec66dce4317a; no voters:
I20260808 12:56:44.036629 28337 raft_consensus.cc:2781] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:56:44.036774 28337 raft_consensus.cc:482] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:56:44.036902 28337 raft_consensus.cc:3037] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:44.039356 28337 raft_consensus.cc:504] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:44.039975 28337 leader_election.cc:288] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [CANDIDATE]: Term 1 election: Requested vote from peers 77541b5634d7460c9a6147298588cd0a (127.24.167.1:42153), 599699167f5f402db480c097187ff662 (127.24.167.2:44625)
I20260808 12:56:44.040470 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349252894a7f40c29a4275787581c0ad" candidate_uuid: "e2daa90edaa84c1c915fec66dce4317a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599699167f5f402db480c097187ff662"
I20260808 12:56:44.040565 27828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349252894a7f40c29a4275787581c0ad" candidate_uuid: "e2daa90edaa84c1c915fec66dce4317a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77541b5634d7460c9a6147298588cd0a"
I20260808 12:56:44.040726 28024 raft_consensus.cc:3037] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:44.040745 27828 raft_consensus.cc:3037] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:44.043110 27828 raft_consensus.cc:2454] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2daa90edaa84c1c915fec66dce4317a in term 1.
I20260808 12:56:44.043161 28024 raft_consensus.cc:2454] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2daa90edaa84c1c915fec66dce4317a in term 1.
I20260808 12:56:44.043527 28127 leader_election.cc:302] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77541b5634d7460c9a6147298588cd0a, e2daa90edaa84c1c915fec66dce4317a; no voters:
I20260808 12:56:44.043800 28337 raft_consensus.cc:2781] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:44.044235 28337 raft_consensus.cc:686] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [term 1 LEADER]: Becoming Leader. State: Replica: e2daa90edaa84c1c915fec66dce4317a, State: Running, Role: LEADER
I20260808 12:56:44.044814 28337 consensus_queue.cc:237] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:44.048344 28367 tablet_replica.cc:439] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a: TxnStatusTablet state changed. Reason: New leader e2daa90edaa84c1c915fec66dce4317a. Latest consensus state: current_term: 1 leader_uuid: "e2daa90edaa84c1c915fec66dce4317a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } }
I20260808 12:56:44.048779 28367 tablet_replica.cc:442] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a: This TxnStatusTablet replica's current role is: LEADER
I20260808 12:56:44.049470 28370 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 12:56:44.049731 28370 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 12:56:44.049974 27616 catalog_manager.cc:5495] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a reported cstate change: term changed from 0 to 1, leader changed from <none> to e2daa90edaa84c1c915fec66dce4317a (127.24.167.3). New cstate: current_term: 1 leader_uuid: "e2daa90edaa84c1c915fec66dce4317a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } health_report { overall_health: HEALTHY } } }
I20260808 12:56:44.069278 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:44.062894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.24.167.3:34183)
0808 12:56:44.063074 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:44.063078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:44.063129 (+ 51us) server_negotiation.cc:407] Connection header received
0808 12:56:44.063470 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:44.063474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:44.063549 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:44.063664 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:56:44.064173 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.065054 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.067026 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.067493 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.067590 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:44.068125 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:56:44.068681 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:56:44.068789 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:44.068959 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 12:56:44.069031 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 12:56:44.069422 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:44.062787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:34183 (local address 127.0.0.1:57634)
0808 12:56:44.063051 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:44.063072 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:44.063244 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:44.063725 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:44.063730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:44.063744 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 12:56:44.063978 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:44.063990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.065195 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.065202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:44.066872 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:44.066888 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.067668 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.067675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:44.067862 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:44.067941 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 12:56:44.068802 (+ 861us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 12:56:44.068830 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 12:56:44.068930 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 12:56:44.069100 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 12:56:44.078819 27828 raft_consensus.cc:1261] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a [term 1 FOLLOWER]: Refusing update from remote peer e2daa90edaa84c1c915fec66dce4317a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:56:44.078888 28024 raft_consensus.cc:1261] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Refusing update from remote peer e2daa90edaa84c1c915fec66dce4317a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:56:44.079779 28367 consensus_queue.cc:1035] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [LEADER]: Connected to new peer: Peer: permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:56:44.080269 28337 consensus_queue.cc:1035] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a [LEADER]: Connected to new peer: Peer: permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:56:44.094118 28335 tablet_replica.cc:439] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a: TxnStatusTablet state changed. Reason: New leader e2daa90edaa84c1c915fec66dce4317a. Latest consensus state: current_term: 1 leader_uuid: "e2daa90edaa84c1c915fec66dce4317a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } }
I20260808 12:56:44.094487 28335 tablet_replica.cc:442] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:56:44.097594 28334 tablet_replica.cc:439] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662: TxnStatusTablet state changed. Reason: New leader e2daa90edaa84c1c915fec66dce4317a. Latest consensus state: current_term: 1 leader_uuid: "e2daa90edaa84c1c915fec66dce4317a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } }
I20260808 12:56:44.097899 28334 tablet_replica.cc:442] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:56:44.098342 28373 mvcc.cc:205] Tried to move back new op lower bound from 7316249821495947264 to 7316249821371719680. Current Snapshot: MvccSnapshot[applied={T|T < 7316249821495947264}]
I20260808 12:56:44.109243 28367 tablet_replica.cc:439] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2daa90edaa84c1c915fec66dce4317a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } }
I20260808 12:56:44.109570 28367 tablet_replica.cc:442] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a: This TxnStatusTablet replica's current role is: LEADER
I20260808 12:56:44.113540 28367 tablet_replica.cc:439] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2daa90edaa84c1c915fec66dce4317a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } }
I20260808 12:56:44.113864 28367 tablet_replica.cc:442] T 349252894a7f40c29a4275787581c0ad P e2daa90edaa84c1c915fec66dce4317a: This TxnStatusTablet replica's current role is: LEADER
I20260808 12:56:44.120146 28334 tablet_replica.cc:439] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2daa90edaa84c1c915fec66dce4317a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } }
I20260808 12:56:44.120507 28334 tablet_replica.cc:442] T 349252894a7f40c29a4275787581c0ad P 599699167f5f402db480c097187ff662: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:56:44.124200 28335 tablet_replica.cc:439] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2daa90edaa84c1c915fec66dce4317a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } } }
I20260808 12:56:44.124605 28335 tablet_replica.cc:442] T 349252894a7f40c29a4275787581c0ad P 77541b5634d7460c9a6147298588cd0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:56:44.128227 28378 mvcc.cc:205] Tried to move back new op lower bound from 7316249821495947264 to 7316249821371719680. Current Snapshot: MvccSnapshot[applied={T|T < 7316249821495947264}]
I20260808 12:56:44.135970 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:44.117008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43994 (local address 127.24.167.1:42153)
0808 12:56:44.117231 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:44.117241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:44.129231 (+ 11990us) server_negotiation.cc:407] Connection header received
0808 12:56:44.129756 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:44.129762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:44.129872 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:44.130014 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:56:44.130797 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.131643 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.133770 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.134161 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.134309 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:44.134865 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:56:44.135373 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:56:44.135552 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:44.135619 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 12:56:44.135704 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 12:56:44.150943 28335 consensus_queue.cc:1035] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:56:44.154783 28392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:44.147214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:34183 (local address 127.24.167.1:45085)
0808 12:56:44.147963 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:44.147979 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:44.148130 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:44.148504 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:44.148507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:44.148517 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 12:56:44.148809 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:44.148817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.150328 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.150334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:44.152083 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:44.152100 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.153309 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.153316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:44.153493 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:44.153573 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 12:56:44.154110 (+ 537us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 12:56:44.154122 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 12:56:44.154194 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 12:56:44.154353 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":424,"threads_started":1}
I20260808 12:56:44.155385 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:44.147274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:45085 (local address 127.24.167.3:34183)
0808 12:56:44.147495 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:44.147502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:44.148085 (+ 583us) server_negotiation.cc:407] Connection header received
0808 12:56:44.148273 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:44.148279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:44.148366 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:44.148467 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:56:44.148984 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.150159 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.152241 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.152995 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.153119 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:44.153818 (+ 699us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:56:44.153971 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:56:44.154057 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:44.155017 (+ 960us) server_negotiation.cc:299] Negotiation successful
0808 12:56:44.155113 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 12:56:44.159710 28335 consensus_queue.cc:1035] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:56:44.200176 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:44.191873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:43859 (local address 127.24.167.1:42153)
0808 12:56:44.192342 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:44.192348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:44.194256 (+ 1908us) server_negotiation.cc:407] Connection header received
0808 12:56:44.194506 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:44.194511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:44.194608 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:44.194720 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:56:44.195316 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.196198 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.198198 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.198710 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.198842 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:44.199306 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:56:44.199477 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:56:44.199566 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:44.199764 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 12:56:44.199846 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343}
I20260808 12:56:44.200551 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:44.191920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:42153 (local address 127.24.167.3:43859)
0808 12:56:44.194150 (+ 2230us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:44.194168 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:44.194328 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:44.194730 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:44.194735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:44.194747 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 12:56:44.195156 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:44.195169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.196332 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.196338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:44.198067 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:44.198085 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:44.198917 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:44.198924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:44.199065 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:44.199121 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 12:56:44.199626 (+ 505us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 12:56:44.199643 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 12:56:44.199744 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 12:56:44.199882 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2034,"thread_start_us":1843,"threads_started":1}
I20260808 12:56:44.250896 27616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:56:44.251255 27616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:56:44.255539 27616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3: Sending DeleteTablet for 3 replicas of tablet f7beb321ec614ac3b542d11634486b4a
I20260808 12:56:44.256924 28211 tablet_service.cc:1514] Processing DeleteTablet for tablet f7beb321ec614ac3b542d11634486b4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:44 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:57632
I20260808 12:56:44.257957 28405 tablet_replica.cc:330] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a: stopping tablet replica
I20260808 12:56:44.257972 27991 tablet_service.cc:1514] Processing DeleteTablet for tablet f7beb321ec614ac3b542d11634486b4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:44 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:46566
I20260808 12:56:44.258430 28405 raft_consensus.cc:2229] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:56:44.258860 28405 raft_consensus.cc:2258] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:44.259150 27802 tablet_service.cc:1514] Processing DeleteTablet for tablet f7beb321ec614ac3b542d11634486b4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:44 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:43986
I20260808 12:56:44.260854 28405 ts_tablet_manager.cc:1905] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:44.263883 27616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:56:44.264587 27616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:56:44.272281 27799 tablet_service.cc:1467] Processing CreateTablet for tablet 842322c338f347afad63999feab1ce5f (DEFAULT_TABLE table=test-table [id=a23570ac1bfc4357a5c4ce49aa37c557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:44.272765 28405 ts_tablet_manager.cc:1918] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 12:56:44.272893 27799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 842322c338f347afad63999feab1ce5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:44.272953 28405 log.cc:1196] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/f7beb321ec614ac3b542d11634486b4a
I20260808 12:56:44.272928 28211 tablet_service.cc:1467] Processing CreateTablet for tablet 842322c338f347afad63999feab1ce5f (DEFAULT_TABLE table=test-table [id=a23570ac1bfc4357a5c4ce49aa37c557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:44.273389 28211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 842322c338f347afad63999feab1ce5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:44.273605 28405 ts_tablet_manager.cc:1939] T f7beb321ec614ac3b542d11634486b4a P e2daa90edaa84c1c915fec66dce4317a: Deleting consensus metadata
I20260808 12:56:44.275635 27601 catalog_manager.cc:4841] TS e2daa90edaa84c1c915fec66dce4317a (127.24.167.3:34183): tablet f7beb321ec614ac3b542d11634486b4a (table test-table [id=cdbb3da180134bc9967eaba58fb70367]) successfully deleted
I20260808 12:56:44.278956 27990 tablet_service.cc:1467] Processing CreateTablet for tablet 842322c338f347afad63999feab1ce5f (DEFAULT_TABLE table=test-table [id=a23570ac1bfc4357a5c4ce49aa37c557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:44.279526 27990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 842322c338f347afad63999feab1ce5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:44.294471 28330 tablet_bootstrap.cc:492] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a: Bootstrap starting.
I20260808 12:56:44.296550 28330 tablet_bootstrap.cc:654] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:44.298727 28329 tablet_bootstrap.cc:492] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a: Bootstrap starting.
I20260808 12:56:44.300649 28329 tablet_bootstrap.cc:654] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:44.303927 28328 tablet_bootstrap.cc:492] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662: Bootstrap starting.
I20260808 12:56:44.305656 28328 tablet_bootstrap.cc:654] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:44.311513 28329 tablet_bootstrap.cc:492] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a: No bootstrap required, opened a new log
I20260808 12:56:44.311699 28329 ts_tablet_manager.cc:1397] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20260808 12:56:44.312584 28329 raft_consensus.cc:348] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:44.312902 28329 raft_consensus.cc:374] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:44.312978 28329 raft_consensus.cc:729] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77541b5634d7460c9a6147298588cd0a, State: Initialized, Role: FOLLOWER
I20260808 12:56:44.313256 28329 consensus_queue.cc:260] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:44.314791 28330 tablet_bootstrap.cc:492] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a: No bootstrap required, opened a new log
I20260808 12:56:44.314901 28329 ts_tablet_manager.cc:1428] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 12:56:44.314996 28330 ts_tablet_manager.cc:1397] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.004s
I20260808 12:56:44.315822 28330 raft_consensus.cc:348] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:44.316087 28330 raft_consensus.cc:374] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:44.316177 28330 raft_consensus.cc:729] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2daa90edaa84c1c915fec66dce4317a, State: Initialized, Role: FOLLOWER
I20260808 12:56:44.316412 28330 consensus_queue.cc:260] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:44.317085 28330 ts_tablet_manager.cc:1428] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260808 12:56:44.318751 28328 tablet_bootstrap.cc:492] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662: No bootstrap required, opened a new log
I20260808 12:56:44.318948 28328 ts_tablet_manager.cc:1397] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20260808 12:56:44.319842 28328 raft_consensus.cc:348] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:44.320113 28328 raft_consensus.cc:374] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:44.320211 28328 raft_consensus.cc:729] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599699167f5f402db480c097187ff662, State: Initialized, Role: FOLLOWER
I20260808 12:56:44.320458 28328 consensus_queue.cc:260] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:44.321051 28328 ts_tablet_manager.cc:1428] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260808 12:56:44.326915 28407 tablet_replica.cc:330] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662: stopping tablet replica
I20260808 12:56:44.326333 28408 tablet_replica.cc:330] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a: stopping tablet replica
I20260808 12:56:44.327541 28408 raft_consensus.cc:2229] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:56:44.328045 28407 raft_consensus.cc:2229] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:56:44.328809 28408 raft_consensus.cc:2258] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:44.329227 28407 raft_consensus.cc:2258] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:44.332710 28408 ts_tablet_manager.cc:1905] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:44.333586 28407 ts_tablet_manager.cc:1905] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:44.344981 28408 ts_tablet_manager.cc:1918] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 12:56:44.344981 28407 ts_tablet_manager.cc:1918] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 12:56:44.345196 28408 log.cc:1196] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/f7beb321ec614ac3b542d11634486b4a
I20260808 12:56:44.345329 28407 log.cc:1196] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/f7beb321ec614ac3b542d11634486b4a
I20260808 12:56:44.345806 28408 ts_tablet_manager.cc:1939] T f7beb321ec614ac3b542d11634486b4a P 77541b5634d7460c9a6147298588cd0a: Deleting consensus metadata
I20260808 12:56:44.345933 28407 ts_tablet_manager.cc:1939] T f7beb321ec614ac3b542d11634486b4a P 599699167f5f402db480c097187ff662: Deleting consensus metadata
I20260808 12:56:44.347774 27597 catalog_manager.cc:4841] TS 599699167f5f402db480c097187ff662 (127.24.167.2:44625): tablet f7beb321ec614ac3b542d11634486b4a (table test-table [id=cdbb3da180134bc9967eaba58fb70367]) successfully deleted
I20260808 12:56:44.347751 27596 catalog_manager.cc:4841] TS 77541b5634d7460c9a6147298588cd0a (127.24.167.1:42153): tablet f7beb321ec614ac3b542d11634486b4a (table test-table [id=cdbb3da180134bc9967eaba58fb70367]) successfully deleted
I20260808 12:56:44.466063 28335 raft_consensus.cc:482] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:56:44.466324 28335 raft_consensus.cc:504] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:44.467195 28335 leader_election.cc:288] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 599699167f5f402db480c097187ff662 (127.24.167.2:44625), e2daa90edaa84c1c915fec66dce4317a (127.24.167.3:34183)
I20260808 12:56:44.467878 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842322c338f347afad63999feab1ce5f" candidate_uuid: "77541b5634d7460c9a6147298588cd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599699167f5f402db480c097187ff662" is_pre_election: true
I20260808 12:56:44.467967 28241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842322c338f347afad63999feab1ce5f" candidate_uuid: "77541b5634d7460c9a6147298588cd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2daa90edaa84c1c915fec66dce4317a" is_pre_election: true
I20260808 12:56:44.468180 28024 raft_consensus.cc:2454] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77541b5634d7460c9a6147298588cd0a in term 0.
I20260808 12:56:44.468264 28241 raft_consensus.cc:2454] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77541b5634d7460c9a6147298588cd0a in term 0.
I20260808 12:56:44.468786 27742 leader_election.cc:302] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77541b5634d7460c9a6147298588cd0a, e2daa90edaa84c1c915fec66dce4317a; no voters:
I20260808 12:56:44.469108 28335 raft_consensus.cc:2781] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:56:44.469218 28335 raft_consensus.cc:482] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:56:44.469327 28335 raft_consensus.cc:3037] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:44.472612 28335 raft_consensus.cc:504] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:44.473829 28241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842322c338f347afad63999feab1ce5f" candidate_uuid: "77541b5634d7460c9a6147298588cd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2daa90edaa84c1c915fec66dce4317a"
I20260808 12:56:44.473834 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842322c338f347afad63999feab1ce5f" candidate_uuid: "77541b5634d7460c9a6147298588cd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599699167f5f402db480c097187ff662"
I20260808 12:56:44.474071 28241 raft_consensus.cc:3037] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:44.474074 28024 raft_consensus.cc:3037] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:44.477329 28024 raft_consensus.cc:2454] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77541b5634d7460c9a6147298588cd0a in term 1.
I20260808 12:56:44.477733 28241 raft_consensus.cc:2454] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77541b5634d7460c9a6147298588cd0a in term 1.
I20260808 12:56:44.478292 27742 leader_election.cc:302] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77541b5634d7460c9a6147298588cd0a, e2daa90edaa84c1c915fec66dce4317a; no voters:
I20260808 12:56:44.478672 28340 raft_consensus.cc:2781] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:44.478853 28340 raft_consensus.cc:686] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [term 1 LEADER]: Becoming Leader. State: Replica: 77541b5634d7460c9a6147298588cd0a, State: Running, Role: LEADER
I20260808 12:56:44.479167 28340 consensus_queue.cc:237] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:44.479982 28335 leader_election.cc:288] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [CANDIDATE]: Term 1 election: Requested vote from peers 599699167f5f402db480c097187ff662 (127.24.167.2:44625), e2daa90edaa84c1c915fec66dce4317a (127.24.167.3:34183)
I20260808 12:56:44.482110 27616 catalog_manager.cc:5495] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 77541b5634d7460c9a6147298588cd0a (127.24.167.1). New cstate: current_term: 1 leader_uuid: "77541b5634d7460c9a6147298588cd0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } health_report { overall_health: UNKNOWN } } }
I20260808 12:56:44.557967 28241 raft_consensus.cc:1261] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Refusing update from remote peer 77541b5634d7460c9a6147298588cd0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:56:44.557988 28024 raft_consensus.cc:1261] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Refusing update from remote peer 77541b5634d7460c9a6147298588cd0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:56:44.558693 28335 consensus_queue.cc:1035] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:56:44.559224 28402 consensus_queue.cc:1035] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 12:56:44.594519 28378 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 12:56:44.598528 28374 mvcc.cc:205] Tried to move back new op lower bound from 7316249823461228544 to 7316249823149633536. Current Snapshot: MvccSnapshot[applied={T|T < 7316249823461228544}]
I20260808 12:56:46.207700 27616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:56:46.208055 27616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:56:46.213807 27616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3: Sending DeleteTablet for 3 replicas of tablet 842322c338f347afad63999feab1ce5f
I20260808 12:56:46.214766 27991 tablet_service.cc:1514] Processing DeleteTablet for tablet 842322c338f347afad63999feab1ce5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:46 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:46566
I20260808 12:56:46.214771 27802 tablet_service.cc:1514] Processing DeleteTablet for tablet 842322c338f347afad63999feab1ce5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:46 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:43986
I20260808 12:56:46.215189 28211 tablet_service.cc:1514] Processing DeleteTablet for tablet 842322c338f347afad63999feab1ce5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:46 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:57632
I20260808 12:56:46.227345 28448 tablet_replica.cc:330] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a: stopping tablet replica
I20260808 12:56:46.228040 28451 tablet_replica.cc:330] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a: stopping tablet replica
I20260808 12:56:46.228504 28451 raft_consensus.cc:2229] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:56:46.229008 28448 raft_consensus.cc:2229] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:56:46.229609 28448 raft_consensus.cc:2258] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:46.229985 28451 raft_consensus.cc:2258] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:46.231554 28451 ts_tablet_manager.cc:1905] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:46.234333 28448 ts_tablet_manager.cc:1905] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:46.237619 28450 tablet_replica.cc:330] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662: stopping tablet replica
I20260808 12:56:46.237831 28450 raft_consensus.cc:2229] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:56:46.238106 28450 raft_consensus.cc:2258] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:46.241866 28448 ts_tablet_manager.cc:1918] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 12:56:46.242067 28448 log.cc:1196] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/842322c338f347afad63999feab1ce5f
I20260808 12:56:46.242347 28451 ts_tablet_manager.cc:1918] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 12:56:46.242569 28451 log.cc:1196] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/842322c338f347afad63999feab1ce5f
I20260808 12:56:46.242626 28448 ts_tablet_manager.cc:1939] T 842322c338f347afad63999feab1ce5f P 77541b5634d7460c9a6147298588cd0a: Deleting consensus metadata
I20260808 12:56:46.243007 28451 ts_tablet_manager.cc:1939] T 842322c338f347afad63999feab1ce5f P e2daa90edaa84c1c915fec66dce4317a: Deleting consensus metadata
I20260808 12:56:46.243933 27596 catalog_manager.cc:4841] TS 77541b5634d7460c9a6147298588cd0a (127.24.167.1:42153): tablet 842322c338f347afad63999feab1ce5f (table test-table [id=a23570ac1bfc4357a5c4ce49aa37c557]) successfully deleted
I20260808 12:56:46.244071 27601 catalog_manager.cc:4841] TS e2daa90edaa84c1c915fec66dce4317a (127.24.167.3:34183): tablet 842322c338f347afad63999feab1ce5f (table test-table [id=a23570ac1bfc4357a5c4ce49aa37c557]) successfully deleted
I20260808 12:56:46.244499 28450 ts_tablet_manager.cc:1905] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:46.250838 28450 ts_tablet_manager.cc:1918] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 12:56:46.251003 28450 log.cc:1196] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/842322c338f347afad63999feab1ce5f
I20260808 12:56:46.251473 28450 ts_tablet_manager.cc:1939] T 842322c338f347afad63999feab1ce5f P 599699167f5f402db480c097187ff662: Deleting consensus metadata
I20260808 12:56:46.252710 27597 catalog_manager.cc:4841] TS 599699167f5f402db480c097187ff662 (127.24.167.2:44625): tablet 842322c338f347afad63999feab1ce5f (table test-table [id=a23570ac1bfc4357a5c4ce49aa37c557]) successfully deleted
Aug 08 12:56:46 dist-test-slave-tk9z krb5kdc[27554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193806, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 12:56:46 dist-test-slave-tk9z krb5kdc[27554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193806, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:46.284703 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:46.266300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.24.167.62:39171)
0808 12:56:46.267045 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:46.267051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:46.267199 (+ 148us) server_negotiation.cc:407] Connection header received
0808 12:56:46.267471 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:46.267476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:46.267541 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:46.267685 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:46.269394 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:46.270647 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:46.272544 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:46.273173 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:46.273334 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:46.278119 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:46.278125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:46.278129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:46.278163 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:46.282049 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:46.282696 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:46.282701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:46.282703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:46.282953 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:46.283279 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:46.283284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:46.283287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:46.283639 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:46.283893 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:46.284099 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 12:56:46.284296 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":348,"threads_started":1}
I20260808 12:56:46.288919 27616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:56:46.289636 27616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:56:46.297246 27802 tablet_service.cc:1467] Processing CreateTablet for tablet 0c27334b64334512bb0912d939ea5adf (DEFAULT_TABLE table=test-table [id=c34239ea5a84494c9fb06a88da3a16f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:46.297485 27991 tablet_service.cc:1467] Processing CreateTablet for tablet 0c27334b64334512bb0912d939ea5adf (DEFAULT_TABLE table=test-table [id=c34239ea5a84494c9fb06a88da3a16f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:46.297711 27802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c27334b64334512bb0912d939ea5adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:46.297705 28211 tablet_service.cc:1467] Processing CreateTablet for tablet 0c27334b64334512bb0912d939ea5adf (DEFAULT_TABLE table=test-table [id=c34239ea5a84494c9fb06a88da3a16f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:46.297923 27991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c27334b64334512bb0912d939ea5adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:46.298273 28211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c27334b64334512bb0912d939ea5adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:46.305011 28472 tablet_bootstrap.cc:492] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a: Bootstrap starting.
I20260808 12:56:46.306169 28473 tablet_bootstrap.cc:492] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a: Bootstrap starting.
I20260808 12:56:46.309278 28473 tablet_bootstrap.cc:654] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:46.309330 28472 tablet_bootstrap.cc:654] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:46.316256 28474 tablet_bootstrap.cc:492] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662: Bootstrap starting.
I20260808 12:56:46.316365 28472 tablet_bootstrap.cc:492] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a: No bootstrap required, opened a new log
I20260808 12:56:46.316496 28473 tablet_bootstrap.cc:492] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a: No bootstrap required, opened a new log
I20260808 12:56:46.316620 28472 ts_tablet_manager.cc:1397] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260808 12:56:46.316701 28473 ts_tablet_manager.cc:1397] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260808 12:56:46.317792 28473 raft_consensus.cc:348] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.318128 28473 raft_consensus.cc:374] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:46.317924 28472 raft_consensus.cc:348] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.318230 28472 raft_consensus.cc:374] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:46.318236 28473 raft_consensus.cc:729] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77541b5634d7460c9a6147298588cd0a, State: Initialized, Role: FOLLOWER
I20260808 12:56:46.318336 28472 raft_consensus.cc:729] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2daa90edaa84c1c915fec66dce4317a, State: Initialized, Role: FOLLOWER
I20260808 12:56:46.318571 28473 consensus_queue.cc:260] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.318635 28472 consensus_queue.cc:260] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.319232 28474 tablet_bootstrap.cc:654] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:46.319353 28472 ts_tablet_manager.cc:1428] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 12:56:46.322551 28473 ts_tablet_manager.cc:1428] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260808 12:56:46.323101 28474 tablet_bootstrap.cc:492] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662: No bootstrap required, opened a new log
I20260808 12:56:46.323335 28474 ts_tablet_manager.cc:1397] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260808 12:56:46.324453 28474 raft_consensus.cc:348] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.324837 28474 raft_consensus.cc:374] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:46.325035 28474 raft_consensus.cc:729] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599699167f5f402db480c097187ff662, State: Initialized, Role: FOLLOWER
I20260808 12:56:46.325302 28474 consensus_queue.cc:260] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.326494 28474 ts_tablet_manager.cc:1428] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 12:56:46.383138 28438 raft_consensus.cc:482] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:56:46.383460 28438 raft_consensus.cc:504] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.384326 28438 leader_election.cc:288] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 599699167f5f402db480c097187ff662 (127.24.167.2:44625), e2daa90edaa84c1c915fec66dce4317a (127.24.167.3:34183)
I20260808 12:56:46.384891 28241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c27334b64334512bb0912d939ea5adf" candidate_uuid: "77541b5634d7460c9a6147298588cd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2daa90edaa84c1c915fec66dce4317a" is_pre_election: true
I20260808 12:56:46.384966 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c27334b64334512bb0912d939ea5adf" candidate_uuid: "77541b5634d7460c9a6147298588cd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599699167f5f402db480c097187ff662" is_pre_election: true
I20260808 12:56:46.385330 28241 raft_consensus.cc:2454] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77541b5634d7460c9a6147298588cd0a in term 0.
I20260808 12:56:46.385329 28022 raft_consensus.cc:2454] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77541b5634d7460c9a6147298588cd0a in term 0.
I20260808 12:56:46.385993 27742 leader_election.cc:302] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 599699167f5f402db480c097187ff662, 77541b5634d7460c9a6147298588cd0a, e2daa90edaa84c1c915fec66dce4317a; no voters:
I20260808 12:56:46.386348 28438 raft_consensus.cc:2781] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:56:46.386591 28438 raft_consensus.cc:482] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:56:46.386770 28438 raft_consensus.cc:3037] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:46.389861 28438 raft_consensus.cc:504] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.390555 28438 leader_election.cc:288] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [CANDIDATE]: Term 1 election: Requested vote from peers 599699167f5f402db480c097187ff662 (127.24.167.2:44625), e2daa90edaa84c1c915fec66dce4317a (127.24.167.3:34183)
I20260808 12:56:46.391476 28241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c27334b64334512bb0912d939ea5adf" candidate_uuid: "77541b5634d7460c9a6147298588cd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2daa90edaa84c1c915fec66dce4317a"
I20260808 12:56:46.391531 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c27334b64334512bb0912d939ea5adf" candidate_uuid: "77541b5634d7460c9a6147298588cd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599699167f5f402db480c097187ff662"
I20260808 12:56:46.391911 28241 raft_consensus.cc:3037] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:46.391911 28022 raft_consensus.cc:3037] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:46.396260 28241 raft_consensus.cc:2454] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77541b5634d7460c9a6147298588cd0a in term 1.
I20260808 12:56:46.396312 28022 raft_consensus.cc:2454] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77541b5634d7460c9a6147298588cd0a in term 1.
I20260808 12:56:46.397002 27742 leader_election.cc:302] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77541b5634d7460c9a6147298588cd0a, e2daa90edaa84c1c915fec66dce4317a; no voters:
I20260808 12:56:46.397415 28438 raft_consensus.cc:2781] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:46.397599 28438 raft_consensus.cc:686] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [term 1 LEADER]: Becoming Leader. State: Replica: 77541b5634d7460c9a6147298588cd0a, State: Running, Role: LEADER
I20260808 12:56:46.398007 28438 consensus_queue.cc:237] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.401684 27615 catalog_manager.cc:5495] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 77541b5634d7460c9a6147298588cd0a (127.24.167.1). New cstate: current_term: 1 leader_uuid: "77541b5634d7460c9a6147298588cd0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } health_report { overall_health: UNKNOWN } } }
I20260808 12:56:46.447256 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:46.441177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.24.167.1:42153)
0808 12:56:46.441963 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:46.441967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:46.441987 (+ 20us) server_negotiation.cc:407] Connection header received
0808 12:56:46.442039 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:46.442042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:46.442101 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:46.442185 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:56:46.442848 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:46.443829 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:46.445432 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:46.445956 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:46.446058 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:46.446576 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:56:46.446677 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:56:46.446748 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:46.446889 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 12:56:46.446957 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":541,"threads_started":1}
I20260808 12:56:46.457371 28241 raft_consensus.cc:1261] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Refusing update from remote peer 77541b5634d7460c9a6147298588cd0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:56:46.457465 28022 raft_consensus.cc:1261] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Refusing update from remote peer 77541b5634d7460c9a6147298588cd0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:56:46.458032 28438 consensus_queue.cc:1035] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:56:46.458482 28435 consensus_queue.cc:1035] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:56:46.511368 27615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:56:46.511714 27615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:56:46.514812 27615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3: Sending DeleteTablet for 3 replicas of tablet 0c27334b64334512bb0912d939ea5adf
I20260808 12:56:46.515836 27802 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c27334b64334512bb0912d939ea5adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:46 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:43986
I20260808 12:56:46.515880 27991 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c27334b64334512bb0912d939ea5adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:46 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:46566
I20260808 12:56:46.516160 28448 tablet_replica.cc:330] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a: stopping tablet replica
I20260808 12:56:46.516577 28450 tablet_replica.cc:330] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662: stopping tablet replica
I20260808 12:56:46.516636 28448 raft_consensus.cc:2229] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:56:46.516955 28450 raft_consensus.cc:2229] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:56:46.517014 27615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 12:56:46.517182 28448 raft_consensus.cc:2258] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:46.517323 28450 raft_consensus.cc:2258] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:46.517926 28211 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c27334b64334512bb0912d939ea5adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:46 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:57632
W20260808 12:56:46.517984 27615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:56:46.518358 28451 tablet_replica.cc:330] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a: stopping tablet replica
I20260808 12:56:46.518718 28451 raft_consensus.cc:2229] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:56:46.518862 28448 ts_tablet_manager.cc:1905] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:46.519027 28451 raft_consensus.cc:2258] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:46.519317 28450 ts_tablet_manager.cc:1905] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:46.520977 28451 ts_tablet_manager.cc:1905] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:46.526988 27802 tablet_service.cc:1467] Processing CreateTablet for tablet ec54aa798f784a8db4fca60582d356dd (DEFAULT_TABLE table=test-table [id=f0a1f7c09f62444bb163ea6024511155]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:46.527413 28448 ts_tablet_manager.cc:1918] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 12:56:46.527606 27802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec54aa798f784a8db4fca60582d356dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:46.527653 27991 tablet_service.cc:1467] Processing CreateTablet for tablet ec54aa798f784a8db4fca60582d356dd (DEFAULT_TABLE table=test-table [id=f0a1f7c09f62444bb163ea6024511155]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:46.527769 28448 log.cc:1196] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/0c27334b64334512bb0912d939ea5adf
I20260808 12:56:46.528155 27991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec54aa798f784a8db4fca60582d356dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:46.528462 28448 ts_tablet_manager.cc:1939] T 0c27334b64334512bb0912d939ea5adf P 77541b5634d7460c9a6147298588cd0a: Deleting consensus metadata
I20260808 12:56:46.529011 28450 ts_tablet_manager.cc:1918] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 12:56:46.529284 28450 log.cc:1196] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/0c27334b64334512bb0912d939ea5adf
I20260808 12:56:46.529975 28450 ts_tablet_manager.cc:1939] T 0c27334b64334512bb0912d939ea5adf P 599699167f5f402db480c097187ff662: Deleting consensus metadata
I20260808 12:56:46.530036 27596 catalog_manager.cc:4841] TS 77541b5634d7460c9a6147298588cd0a (127.24.167.1:42153): tablet 0c27334b64334512bb0912d939ea5adf (table test-table [id=c34239ea5a84494c9fb06a88da3a16f4]) successfully deleted
I20260808 12:56:46.531430 27597 catalog_manager.cc:4841] TS 599699167f5f402db480c097187ff662 (127.24.167.2:44625): tablet 0c27334b64334512bb0912d939ea5adf (table test-table [id=c34239ea5a84494c9fb06a88da3a16f4]) successfully deleted
I20260808 12:56:46.534679 28451 ts_tablet_manager.cc:1918] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 12:56:46.535054 28451 log.cc:1196] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/0c27334b64334512bb0912d939ea5adf
I20260808 12:56:46.535645 28451 ts_tablet_manager.cc:1939] T 0c27334b64334512bb0912d939ea5adf P e2daa90edaa84c1c915fec66dce4317a: Deleting consensus metadata
I20260808 12:56:46.536005 28474 tablet_bootstrap.cc:492] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662: Bootstrap starting.
I20260808 12:56:46.536815 28211 tablet_service.cc:1467] Processing CreateTablet for tablet ec54aa798f784a8db4fca60582d356dd (DEFAULT_TABLE table=test-table [id=f0a1f7c09f62444bb163ea6024511155]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:56:46.536918 27601 catalog_manager.cc:4841] TS e2daa90edaa84c1c915fec66dce4317a (127.24.167.3:34183): tablet 0c27334b64334512bb0912d939ea5adf (table test-table [id=c34239ea5a84494c9fb06a88da3a16f4]) successfully deleted
I20260808 12:56:46.537536 28211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec54aa798f784a8db4fca60582d356dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:46.538079 28473 tablet_bootstrap.cc:492] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a: Bootstrap starting.
I20260808 12:56:46.539415 28474 tablet_bootstrap.cc:654] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:46.540840 28473 tablet_bootstrap.cc:654] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:46.545156 28474 tablet_bootstrap.cc:492] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662: No bootstrap required, opened a new log
I20260808 12:56:46.545488 28474 ts_tablet_manager.cc:1397] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.003s
I20260808 12:56:46.545632 28473 tablet_bootstrap.cc:492] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a: No bootstrap required, opened a new log
I20260808 12:56:46.546023 28473 ts_tablet_manager.cc:1397] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 12:56:46.547154 28474 raft_consensus.cc:348] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.547631 28474 raft_consensus.cc:374] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:46.547797 28474 raft_consensus.cc:729] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599699167f5f402db480c097187ff662, State: Initialized, Role: FOLLOWER
I20260808 12:56:46.547585 28473 raft_consensus.cc:348] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.548048 28473 raft_consensus.cc:374] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:46.548189 28473 raft_consensus.cc:729] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77541b5634d7460c9a6147298588cd0a, State: Initialized, Role: FOLLOWER
I20260808 12:56:46.548230 28474 consensus_queue.cc:260] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.548671 28472 tablet_bootstrap.cc:492] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a: Bootstrap starting.
I20260808 12:56:46.548491 28473 consensus_queue.cc:260] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.550238 28473 ts_tablet_manager.cc:1428] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 12:56:46.551458 28472 tablet_bootstrap.cc:654] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:46.551427 28474 ts_tablet_manager.cc:1428] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260808 12:56:46.555068 28472 tablet_bootstrap.cc:492] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a: No bootstrap required, opened a new log
I20260808 12:56:46.555328 28472 ts_tablet_manager.cc:1397] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 12:56:46.556224 28472 raft_consensus.cc:348] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.556516 28472 raft_consensus.cc:374] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:46.556685 28472 raft_consensus.cc:729] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2daa90edaa84c1c915fec66dce4317a, State: Initialized, Role: FOLLOWER
I20260808 12:56:46.557056 28472 consensus_queue.cc:260] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.557623 28472 ts_tablet_manager.cc:1428] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260808 12:56:46.588024 28337 raft_consensus.cc:482] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:56:46.588261 28337 raft_consensus.cc:504] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.589010 28337 leader_election.cc:288] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77541b5634d7460c9a6147298588cd0a (127.24.167.1:42153), 599699167f5f402db480c097187ff662 (127.24.167.2:44625)
I20260808 12:56:46.589563 27827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54aa798f784a8db4fca60582d356dd" candidate_uuid: "e2daa90edaa84c1c915fec66dce4317a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77541b5634d7460c9a6147298588cd0a" is_pre_election: true
I20260808 12:56:46.589700 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54aa798f784a8db4fca60582d356dd" candidate_uuid: "e2daa90edaa84c1c915fec66dce4317a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599699167f5f402db480c097187ff662" is_pre_election: true
I20260808 12:56:46.589962 27827 raft_consensus.cc:2454] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2daa90edaa84c1c915fec66dce4317a in term 0.
I20260808 12:56:46.590202 28022 raft_consensus.cc:2454] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2daa90edaa84c1c915fec66dce4317a in term 0.
I20260808 12:56:46.590673 28127 leader_election.cc:302] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77541b5634d7460c9a6147298588cd0a, e2daa90edaa84c1c915fec66dce4317a; no voters:
I20260808 12:56:46.591008 28337 raft_consensus.cc:2781] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:56:46.591224 28337 raft_consensus.cc:482] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:56:46.591369 28337 raft_consensus.cc:3037] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:46.593791 28337 raft_consensus.cc:504] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.594460 28337 leader_election.cc:288] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [CANDIDATE]: Term 1 election: Requested vote from peers 77541b5634d7460c9a6147298588cd0a (127.24.167.1:42153), 599699167f5f402db480c097187ff662 (127.24.167.2:44625)
I20260808 12:56:46.595003 27827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54aa798f784a8db4fca60582d356dd" candidate_uuid: "e2daa90edaa84c1c915fec66dce4317a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77541b5634d7460c9a6147298588cd0a"
I20260808 12:56:46.595244 27827 raft_consensus.cc:3037] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:46.595216 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec54aa798f784a8db4fca60582d356dd" candidate_uuid: "e2daa90edaa84c1c915fec66dce4317a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599699167f5f402db480c097187ff662"
I20260808 12:56:46.595494 28022 raft_consensus.cc:3037] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:46.598492 27827 raft_consensus.cc:2454] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2daa90edaa84c1c915fec66dce4317a in term 1.
I20260808 12:56:46.598830 28022 raft_consensus.cc:2454] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2daa90edaa84c1c915fec66dce4317a in term 1.
I20260808 12:56:46.599083 28127 leader_election.cc:302] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77541b5634d7460c9a6147298588cd0a, e2daa90edaa84c1c915fec66dce4317a; no voters:
I20260808 12:56:46.599406 28337 raft_consensus.cc:2781] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:46.599596 28337 raft_consensus.cc:686] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [term 1 LEADER]: Becoming Leader. State: Replica: e2daa90edaa84c1c915fec66dce4317a, State: Running, Role: LEADER
I20260808 12:56:46.599908 28337 consensus_queue.cc:237] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } }
I20260808 12:56:46.602666 27615 catalog_manager.cc:5495] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a reported cstate change: term changed from 0 to 1, leader changed from <none> to e2daa90edaa84c1c915fec66dce4317a (127.24.167.3). New cstate: current_term: 1 leader_uuid: "e2daa90edaa84c1c915fec66dce4317a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2daa90edaa84c1c915fec66dce4317a" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 34183 } health_report { overall_health: HEALTHY } } }
I20260808 12:56:46.632388 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:46.624669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57644 (local address 127.24.167.3:34183)
0808 12:56:46.625761 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:46.625765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:46.625785 (+ 20us) server_negotiation.cc:407] Connection header received
0808 12:56:46.625842 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:46.625846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:46.625937 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:46.626063 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:56:46.626722 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:46.627814 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:46.629811 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:46.630532 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:46.630689 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:46.631374 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:56:46.631529 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:56:46.631646 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:46.631808 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 12:56:46.631892 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962,"thread_start_us":736,"threads_started":1}
I20260808 12:56:46.649473 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:46.642774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:39023 (local address 127.24.167.3:34183)
0808 12:56:46.642926 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:46.642932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:46.644005 (+ 1073us) server_negotiation.cc:407] Connection header received
0808 12:56:46.644217 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:46.644228 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:46.644280 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:46.644367 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:56:46.644966 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:46.645720 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:46.647515 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:46.648014 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:46.648149 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:46.648664 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:56:46.648754 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:56:46.648849 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:46.648984 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 12:56:46.649047 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 12:56:46.649664 28495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:46.642895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:34183 (local address 127.24.167.3:39023)
0808 12:56:46.643900 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:46.643916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:46.644097 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:46.644379 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:46.644383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:46.644393 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 12:56:46.644718 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:46.644726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:46.645844 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:46.645851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:46.647377 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:46.647389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:46.648229 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:46.648240 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:46.648471 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:46.648507 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 12:56:46.648849 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 12:56:46.648861 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 12:56:46.648965 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 12:56:46.649090 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":685,"threads_started":1}
I20260808 12:56:46.657684 28022 raft_consensus.cc:1261] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Refusing update from remote peer e2daa90edaa84c1c915fec66dce4317a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:56:46.657819 27827 raft_consensus.cc:1261] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a [term 1 FOLLOWER]: Refusing update from remote peer e2daa90edaa84c1c915fec66dce4317a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:56:46.658357 28421 consensus_queue.cc:1035] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [LEADER]: Connected to new peer: Peer: permanent_uuid: "599699167f5f402db480c097187ff662" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:56:46.658793 28337 consensus_queue.cc:1035] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [LEADER]: Connected to new peer: Peer: permanent_uuid: "77541b5634d7460c9a6147298588cd0a" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 12:56:46.680883 28419 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 12:56:48.301571 27615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:56:48.301880 27615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:56:48.307852 27615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 600a14dc4ddb4cabb162562fd78841f3: Sending DeleteTablet for 3 replicas of tablet ec54aa798f784a8db4fca60582d356dd
I20260808 12:56:48.309007 27802 tablet_service.cc:1514] Processing DeleteTablet for tablet ec54aa798f784a8db4fca60582d356dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:48 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:43986
I20260808 12:56:48.309095 27991 tablet_service.cc:1514] Processing DeleteTablet for tablet ec54aa798f784a8db4fca60582d356dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:48 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:46566
I20260808 12:56:48.309118 28211 tablet_service.cc:1514] Processing DeleteTablet for tablet ec54aa798f784a8db4fca60582d356dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:56:48 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:57632
I20260808 12:56:48.310356 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 27718
I20260808 12:56:48.325301 28529 tablet_replica.cc:330] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a: stopping tablet replica
I20260808 12:56:48.325662 28529 raft_consensus.cc:2229] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:56:48.326002 28529 raft_consensus.cc:2258] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:48.326721 28531 tablet_replica.cc:330] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662: stopping tablet replica
I20260808 12:56:48.326995 28531 raft_consensus.cc:2229] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:56:48.327556 28531 raft_consensus.cc:2258] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:48.328887 28529 ts_tablet_manager.cc:1905] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:48.335228 28531 ts_tablet_manager.cc:1905] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:48.335271 28536 tablet_replica.cc:330] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a: stopping tablet replica
I20260808 12:56:48.335816 28536 raft_consensus.cc:2229] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:56:48.336570 28536 raft_consensus.cc:2258] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:56:48.337858 28536 ts_tablet_manager.cc:1905] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:56:48.339066 28529 ts_tablet_manager.cc:1918] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 12:56:48.339265 28529 log.cc:1196] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/ec54aa798f784a8db4fca60582d356dd
I20260808 12:56:48.339792 28529 ts_tablet_manager.cc:1939] T ec54aa798f784a8db4fca60582d356dd P 77541b5634d7460c9a6147298588cd0a: Deleting consensus metadata
I20260808 12:56:48.341161 27596 catalog_manager.cc:4841] TS 77541b5634d7460c9a6147298588cd0a (127.24.167.1:42153): tablet ec54aa798f784a8db4fca60582d356dd (table test-table [id=f0a1f7c09f62444bb163ea6024511155]) successfully deleted
I20260808 12:56:48.342641 28531 ts_tablet_manager.cc:1918] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 12:56:48.342826 28531 log.cc:1196] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/ec54aa798f784a8db4fca60582d356dd
I20260808 12:56:48.343295 28531 ts_tablet_manager.cc:1939] T ec54aa798f784a8db4fca60582d356dd P 599699167f5f402db480c097187ff662: Deleting consensus metadata
I20260808 12:56:48.344585 27597 catalog_manager.cc:4841] TS 599699167f5f402db480c097187ff662 (127.24.167.2:44625): tablet ec54aa798f784a8db4fca60582d356dd (table test-table [id=f0a1f7c09f62444bb163ea6024511155]) successfully deleted
I20260808 12:56:48.345082 28536 ts_tablet_manager.cc:1918] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 12:56:48.345271 28536 log.cc:1196] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/ec54aa798f784a8db4fca60582d356dd
I20260808 12:56:48.345692 28536 ts_tablet_manager.cc:1939] T ec54aa798f784a8db4fca60582d356dd P e2daa90edaa84c1c915fec66dce4317a: Deleting consensus metadata
I20260808 12:56:48.346783 27601 catalog_manager.cc:4841] TS e2daa90edaa84c1c915fec66dce4317a (127.24.167.3:34183): tablet ec54aa798f784a8db4fca60582d356dd (table test-table [id=f0a1f7c09f62444bb163ea6024511155]) successfully deleted
Aug 08 12:56:48 dist-test-slave-tk9z krb5kdc[27554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193806, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:56:48.355433 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:48.312111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.24.167.1:42153)
0808 12:56:48.330876 (+ 18765us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:48.330898 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:48.331438 (+ 540us) server_negotiation.cc:407] Connection header received
0808 12:56:48.331652 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:48.331658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:48.331744 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:48.331864 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:48.333427 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:48.334964 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:48.337980 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:48.338712 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:48.338941 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:48.348604 (+ 9663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:48.348621 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:48.348639 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:48.348699 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:48.352430 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:48.353143 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:48.353150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:48.353153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:48.353340 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:48.353808 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:48.353811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:48.353814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:48.354261 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:48.354651 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:48.354808 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 12:56:48.354990 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18555,"thread_start_us":18435,"threads_started":1}
W20260808 12:56:48.356173 27887 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44014
W20260808 12:56:48.356917 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 27718: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 12:56:48.357199 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 27718
I20260808 12:56:48.392336 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 27902
Aug 08 12:56:48 dist-test-slave-tk9z krb5kdc[27554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193806, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 12:56:48.408775 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:48.393163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46580 (local address 127.24.167.2:44625)
0808 12:56:48.393773 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:48.393777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:48.393795 (+ 18us) server_negotiation.cc:407] Connection header received
0808 12:56:48.393853 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:48.393857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:48.393917 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:48.394040 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:48.395757 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:48.396851 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:48.398075 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:48.398577 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:48.398682 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:48.403106 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:48.403125 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:48.403144 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:48.403182 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:48.406267 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:48.407002 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:48.407010 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:48.407012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:48.407162 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:48.407504 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:48.407507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:48.407509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:48.407836 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:48.408081 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:48.408323 (+ 242us) server_negotiation.cc:299] Negotiation successful
0808 12:56:48.408501 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":359,"threads_started":1}
W20260808 12:56:48.409518 28090 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46580
W20260808 12:56:48.410146 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 27902: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 12:56:48.410315 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 27902
I20260808 12:56:48.441926 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 28105
Aug 08 12:56:48 dist-test-slave-tk9z krb5kdc[27554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193806, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 12:56:48.458112 28542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:48.442759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.24.167.3:34183)
0808 12:56:48.443303 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:48.443307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:48.443324 (+ 17us) server_negotiation.cc:407] Connection header received
0808 12:56:48.443380 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:48.443384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:48.443443 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:48.443548 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:48.444781 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:48.445508 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:48.446907 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:48.447299 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:48.447392 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:48.451994 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:48.452010 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:48.452029 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:48.452064 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:48.455257 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:48.455951 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:48.455959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:48.455961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:48.456101 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:48.456498 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:48.456503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:48.456506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:48.456861 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:48.457214 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:48.457510 (+ 296us) server_negotiation.cc:299] Negotiation successful
0808 12:56:48.457749 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":305,"threads_started":1}
W20260808 12:56:48.458930 28299 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57648
W20260808 12:56:48.459573 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 28105: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 12:56:48.459720 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 28105
I20260808 12:56:48.494184 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 27574
I20260808 12:56:48.495472 27672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:56:48.618875 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 27574
2026-08-08T12:56:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9014 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:56:48 dist-test-slave-tk9z krb5kdc[28547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:56:48 dist-test-slave-tk9z krb5kdc[28547](info): set up 2 sockets
Aug 08 12:56:48 dist-test-slave-tk9z krb5kdc[28547](info): commencing operation
krb5kdc: starting...
W20260808 12:56:49.924523 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.251s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:56:50 dist-test-slave-tk9z krb5kdc[28547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:56:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:56:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 12:56:50.137101 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:37165
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:37165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:50.468432 28564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:50.468719 28564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:50.468813 28564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:50.475975 28564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:56:50.476086 28564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:50.476137 28564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:50.476195 28564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:56:50.476239 28564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:56:50.486431 28564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:37165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:37165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:50.487687 28564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:50.488938 28564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:50.500321 28572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:50.500388 28571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:50.500419 28575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:50.501402 28564 server_base.cc:1034] running on GCE node
I20260808 12:56:50.502101 28564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:50.503504 28564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:50.504709 28564 hybrid_clock.cc:648] HybridClock initialized: now 1786193810504648 us; error 46 us; skew 500 ppm
I20260808 12:56:50.505067 28564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:50 dist-test-slave-tk9z krb5kdc[28547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193810, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:50.508236 28564 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:56:50.509644 28564 webserver.cc:460] Webserver started at http://127.24.167.62:40755/ using document root <none> and password file <none>
I20260808 12:56:50.510298 28564 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:50.510459 28564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:50.510737 28564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:50.513643 28564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "091e38328a404a1ba08092fbcb048dc8"
format_stamp: "Formatted at 2026-08-08 12:56:50 on dist-test-slave-tk9z"
server_key: "d170d57f709c662b14d79e50fb9d404d"
server_key_iv: "cb0fc2cb014d724f6af264b9efe74b68"
server_key_version: "encryptionkey@0"
I20260808 12:56:50.514416 28564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "091e38328a404a1ba08092fbcb048dc8"
format_stamp: "Formatted at 2026-08-08 12:56:50 on dist-test-slave-tk9z"
server_key: "d170d57f709c662b14d79e50fb9d404d"
server_key_iv: "cb0fc2cb014d724f6af264b9efe74b68"
server_key_version: "encryptionkey@0"
I20260808 12:56:50.519650 28564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 12:56:50.523057 28584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:50.524296 28564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 12:56:50.524501 28564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "091e38328a404a1ba08092fbcb048dc8"
format_stamp: "Formatted at 2026-08-08 12:56:50 on dist-test-slave-tk9z"
server_key: "d170d57f709c662b14d79e50fb9d404d"
server_key_iv: "cb0fc2cb014d724f6af264b9efe74b68"
server_key_version: "encryptionkey@0"
I20260808 12:56:50.524710 28564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:50.549688 28564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:50.553591 28564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:50.553889 28564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:50.572619 28564 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:37165
I20260808 12:56:50.572654 28650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:37165 every 8 connection(s)
I20260808 12:56:50.574050 28564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:56:50.576018 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 28564
I20260808 12:56:50.576225 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:56:50.576915 25244 external_mini_cluster.cc:1428] Setting key fb5aff555ab64c013efdb47ad1b76a67
I20260808 12:56:50.578292 28651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 12:56:50 dist-test-slave-tk9z krb5kdc[28547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:50.592298 28651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8: Bootstrap starting.
I20260808 12:56:50.596413 28651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:50.597702 28651 log.cc:824] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8: Log is configured to *not* fsync() on all Append() calls
I20260808 12:56:50.601114 28651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8: No bootstrap required, opened a new log
I20260808 12:56:50.601433 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:50.578988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.24.167.62:37165)
0808 12:56:50.579839 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:50.579854 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:50.579909 (+ 55us) server_negotiation.cc:407] Connection header received
0808 12:56:50.581850 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:50.581877 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:50.582375 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:50.582856 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:50.584204 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:50.586285 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:50.587684 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:50.588718 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:50.588912 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:50.592898 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:50.592908 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:50.592925 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:50.592974 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:50.597000 (+ 4026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:50.597711 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:50.597723 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:50.597732 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:50.597957 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:50.598415 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:50.598421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:50.598424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:50.599153 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:50.599504 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:50.600047 (+ 543us) server_negotiation.cc:299] Negotiation successful
0808 12:56:50.600454 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":342,"threads_started":1}
I20260808 12:56:50.608873 28651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091e38328a404a1ba08092fbcb048dc8" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 37165 } }
I20260808 12:56:50.609256 28651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:50.609365 28651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091e38328a404a1ba08092fbcb048dc8, State: Initialized, Role: FOLLOWER
I20260808 12:56:50.610052 28651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091e38328a404a1ba08092fbcb048dc8" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 37165 } }
I20260808 12:56:50.610350 28651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:56:50.610582 28651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:56:50.610769 28651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:50.612838 28651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091e38328a404a1ba08092fbcb048dc8" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 37165 } }
I20260808 12:56:50.613272 28651 leader_election.cc:302] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 091e38328a404a1ba08092fbcb048dc8; no voters:
I20260808 12:56:50.613821 28651 leader_election.cc:288] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:56:50.613966 28657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:50.614678 28657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [term 1 LEADER]: Becoming Leader. State: Replica: 091e38328a404a1ba08092fbcb048dc8, State: Running, Role: LEADER
I20260808 12:56:50.615267 28657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091e38328a404a1ba08092fbcb048dc8" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 37165 } }
I20260808 12:56:50.615963 28651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:56:50.619477 28658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "091e38328a404a1ba08092fbcb048dc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091e38328a404a1ba08092fbcb048dc8" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 37165 } } }
I20260808 12:56:50.619990 28658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [sys.catalog]: This master's current role is: LEADER
I20260808 12:56:50.620376 28659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 091e38328a404a1ba08092fbcb048dc8. Latest consensus state: current_term: 1 leader_uuid: "091e38328a404a1ba08092fbcb048dc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091e38328a404a1ba08092fbcb048dc8" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 37165 } } }
I20260808 12:56:50.620714 28659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8 [sys.catalog]: This master's current role is: LEADER
I20260808 12:56:50.621610 28669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:56:50.627053 28669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:56:50.636279 28669 catalog_manager.cc:1347] Generated new cluster ID: 9bce5c454a904e25aa37f6d634476bf6
I20260808 12:56:50.636406 28669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:56:50.651479 28669 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:56:50.652840 28669 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:56:50.661991 28669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 091e38328a404a1ba08092fbcb048dc8: Generated new TSK 0
I20260808 12:56:50.662891 28669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 12:56:50.725849 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:37165
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:51.089641 28687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:51.089951 28687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:51.090113 28687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:51.097388 28687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:51.097529 28687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:51.097694 28687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:56:51.107766 28687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:37165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:51.109016 28687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:51.110301 28687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:51.123824 28699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:51.123900 28697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:51.124353 28696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:51.124979 28687 server_base.cc:1034] running on GCE node
I20260808 12:56:51.125566 28687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:51.126518 28687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:51.127736 28687 hybrid_clock.cc:648] HybridClock initialized: now 1786193811127678 us; error 40 us; skew 500 ppm
I20260808 12:56:51.128065 28687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:51 dist-test-slave-tk9z krb5kdc[28547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193811, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:51.131697 28687 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 12:56:51.133236 28687 webserver.cc:460] Webserver started at http://127.24.167.1:41135/ using document root <none> and password file <none>
I20260808 12:56:51.133940 28687 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:51.134084 28687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:51.134445 28687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:51.137339 28687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "e2a1f6e678da40c08e7acb3bd236fb09"
format_stamp: "Formatted at 2026-08-08 12:56:51 on dist-test-slave-tk9z"
server_key: "472c367aeed39902d08b78e9720b0bde"
server_key_iv: "dd39cb0aa4f0ed16c9392a15a55aadd5"
server_key_version: "encryptionkey@0"
I20260808 12:56:51.138157 28687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "e2a1f6e678da40c08e7acb3bd236fb09"
format_stamp: "Formatted at 2026-08-08 12:56:51 on dist-test-slave-tk9z"
server_key: "472c367aeed39902d08b78e9720b0bde"
server_key_iv: "dd39cb0aa4f0ed16c9392a15a55aadd5"
server_key_version: "encryptionkey@0"
I20260808 12:56:51.143393 28687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260808 12:56:51.146725 28708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:51.148039 28687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 12:56:51.148213 28687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "e2a1f6e678da40c08e7acb3bd236fb09"
format_stamp: "Formatted at 2026-08-08 12:56:51 on dist-test-slave-tk9z"
server_key: "472c367aeed39902d08b78e9720b0bde"
server_key_iv: "dd39cb0aa4f0ed16c9392a15a55aadd5"
server_key_version: "encryptionkey@0"
I20260808 12:56:51.148413 28687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:51.163053 28687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:51.167222 28687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:51.167512 28687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:51.168468 28687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:56:51.169857 28687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:51.169960 28687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:51.170094 28687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:51.170166 28687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:51.202720 28687 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:43137
I20260808 12:56:51.202770 28871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:43137 every 8 connection(s)
I20260808 12:56:51.204046 28687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:56:51.210213 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 28687
I20260808 12:56:51.210469 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:56:51.211129 25244 external_mini_cluster.cc:1428] Setting key 6d061c50c4f9b328faa152c3582121f4
Aug 08 12:56:51 dist-test-slave-tk9z krb5kdc[28547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193811, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:51.225090 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:51.206566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:43103 (local address 127.24.167.62:37165)
0808 12:56:51.207038 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:51.207042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:51.207898 (+ 856us) server_negotiation.cc:407] Connection header received
0808 12:56:51.209758 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:51.209764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:51.209872 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:51.209993 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:51.211721 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:51.212624 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:51.214165 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:51.214640 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:51.214780 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:51.218827 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:51.218832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:51.218835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:51.218869 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:51.221735 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:51.222702 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:51.222706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:51.222709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:51.222832 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:51.223283 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:51.223287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:51.223289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:51.223565 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:51.223740 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:51.224469 (+ 729us) server_negotiation.cc:299] Negotiation successful
0808 12:56:51.224706 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":258,"threads_started":1}
I20260808 12:56:51.226084 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:51.206996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:37165 (local address 127.24.167.1:43103)
0808 12:56:51.207729 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:51.207768 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:51.209483 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:51.210243 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:51.210251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:51.210913 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:51.211543 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:51.211561 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:51.212781 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:51.212786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:51.214017 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:51.214030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:51.214887 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:51.214897 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:51.215144 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:51.215857 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:51.215880 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:51.218590 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:51.221891 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:51.221902 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:51.221920 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:51.222569 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:51.222945 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:51.222952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:51.222955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:51.223159 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:51.223698 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:51.223706 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:51.224078 (+ 372us) client_negotiation.cc:769] Sending connection context
0808 12:56:51.224349 (+ 271us) client_negotiation.cc:240] Negotiation successful
0808 12:56:51.224907 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":303,"threads_started":1}
I20260808 12:56:51.227849 28872 heartbeater.cc:347] Connected to a master server at 127.24.167.62:37165
I20260808 12:56:51.228183 28872 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:51.228940 28872 heartbeater.cc:511] Master 127.24.167.62:37165 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
I20260808 12:56:51.231120 28603 ts_manager.cc:194] Registered new tserver with Master: e2a1f6e678da40c08e7acb3bd236fb09 (127.24.167.1:43137)
I20260808 12:56:51.232539 28603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:43103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 12:56:51.257807 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:37165
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:51.601083 28880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:51.601361 28880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:51.601456 28880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:51.608647 28880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:51.608763 28880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:51.608891 28880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:56:51.618733 28880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:37165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:51.619993 28880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:51.621317 28880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:51.634534 28886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:51.634613 28889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:51.634578 28885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:51.635406 28880 server_base.cc:1034] running on GCE node
I20260808 12:56:51.636070 28880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:51.637092 28880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:51.638319 28880 hybrid_clock.cc:648] HybridClock initialized: now 1786193811638279 us; error 55 us; skew 500 ppm
I20260808 12:56:51.638733 28880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:51 dist-test-slave-tk9z krb5kdc[28547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193811, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:51.641942 28880 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 12:56:51.643425 28880 webserver.cc:460] Webserver started at http://127.24.167.2:33075/ using document root <none> and password file <none>
I20260808 12:56:51.644129 28880 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:51.644261 28880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:51.644575 28880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:51.647560 28880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "5e003889898b41598c2be37ef53607b0"
format_stamp: "Formatted at 2026-08-08 12:56:51 on dist-test-slave-tk9z"
server_key: "d21b2518b6cad00713d03fe5858810ff"
server_key_iv: "36c01401227c15eed72285faaedcf051"
server_key_version: "encryptionkey@0"
I20260808 12:56:51.648343 28880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "5e003889898b41598c2be37ef53607b0"
format_stamp: "Formatted at 2026-08-08 12:56:51 on dist-test-slave-tk9z"
server_key: "d21b2518b6cad00713d03fe5858810ff"
server_key_iv: "36c01401227c15eed72285faaedcf051"
server_key_version: "encryptionkey@0"
I20260808 12:56:51.653759 28880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260808 12:56:51.657433 28900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:51.658882 28880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:56:51.659065 28880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "5e003889898b41598c2be37ef53607b0"
format_stamp: "Formatted at 2026-08-08 12:56:51 on dist-test-slave-tk9z"
server_key: "d21b2518b6cad00713d03fe5858810ff"
server_key_iv: "36c01401227c15eed72285faaedcf051"
server_key_version: "encryptionkey@0"
I20260808 12:56:51.659271 28880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:51.687216 28880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:51.691231 28880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:51.691531 28880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:51.692497 28880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:56:51.693862 28880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:51.693964 28880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:51.694108 28880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:51.694180 28880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:51.727313 28880 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:34355
I20260808 12:56:51.727336 29074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:34355 every 8 connection(s)
I20260808 12:56:51.728780 28880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:56:51.738912 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 28880
I20260808 12:56:51.739137 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:56:51.739657 25244 external_mini_cluster.cc:1428] Setting key f8310f329ce0fa2d39fa15cfafa23ad5
Aug 08 12:56:51 dist-test-slave-tk9z krb5kdc[28547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193811, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:51.748996 29078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:51.731481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:47643 (local address 127.24.167.62:37165)
0808 12:56:51.731922 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:51.731926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:51.732931 (+ 1005us) server_negotiation.cc:407] Connection header received
0808 12:56:51.734845 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:51.734849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:51.734926 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:51.735018 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:51.736474 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:51.737194 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:51.738917 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:51.739430 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:51.739549 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:51.743286 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:51.743292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:51.743302 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:51.743333 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:51.745866 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:51.746680 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:51.746685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:51.746688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:51.746824 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:51.747257 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:51.747261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:51.747265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:51.747534 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:51.747686 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:51.748375 (+ 689us) server_negotiation.cc:299] Negotiation successful
0808 12:56:51.748620 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":217,"threads_started":1}
I20260808 12:56:51.749915 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:51.732056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:37165 (local address 127.24.167.2:47643)
0808 12:56:51.732768 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:51.732814 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:51.734568 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:51.735214 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:51.735221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:51.735771 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:51.736323 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:51.736339 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:51.737314 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:51.737318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:51.738769 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:51.738791 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:51.739677 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:51.739686 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:51.739988 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:51.740790 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:51.740814 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:51.743068 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:51.746016 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:51.746025 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:51.746041 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:51.746558 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:51.746942 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:51.746947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:51.746950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:51.747137 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:51.747661 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:51.747669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:51.748019 (+ 350us) client_negotiation.cc:769] Sending connection context
0808 12:56:51.748278 (+ 259us) client_negotiation.cc:240] Negotiation successful
0808 12:56:51.748804 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":301,"threads_started":1}
I20260808 12:56:51.751387 29075 heartbeater.cc:347] Connected to a master server at 127.24.167.62:37165
I20260808 12:56:51.751784 29075 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:51.752691 29075 heartbeater.cc:511] Master 127.24.167.62:37165 requested a full tablet report, sending...
I20260808 12:56:51.754575 28603 ts_manager.cc:194] Registered new tserver with Master: 5e003889898b41598c2be37ef53607b0 (127.24.167.2:34355)
I20260808 12:56:51.755432 28603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:47643
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 12:56:51.789417 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:37165
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:52.128575 29082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:52.128892 29082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:52.129025 29082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:52.136317 29082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:52.136435 29082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:52.136551 29082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:56:52.146312 29082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:37165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:52.147785 29082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:52.149070 29082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:52.162555 29090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:52.162592 29092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:52.163029 29094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:52.163396 29082 server_base.cc:1034] running on GCE node
I20260808 12:56:52.163975 29082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:52.164885 29082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:52.166103 29082 hybrid_clock.cc:648] HybridClock initialized: now 1786193812166059 us; error 52 us; skew 500 ppm
I20260808 12:56:52.166455 29082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:52 dist-test-slave-tk9z krb5kdc[28547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193812, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:52.169570 29082 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 12:56:52.171198 29082 webserver.cc:460] Webserver started at http://127.24.167.3:32837/ using document root <none> and password file <none>
I20260808 12:56:52.171881 29082 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:52.172024 29082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:52.172335 29082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:52.175266 29082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "71d7843a8a424cdf81951041cd817cff"
format_stamp: "Formatted at 2026-08-08 12:56:52 on dist-test-slave-tk9z"
server_key: "5c50efb0199c21a6f23dc9362b5d3c4d"
server_key_iv: "d6f9a4226fa1b9c991966a4f39f8fa60"
server_key_version: "encryptionkey@0"
I20260808 12:56:52.176054 29082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "71d7843a8a424cdf81951041cd817cff"
format_stamp: "Formatted at 2026-08-08 12:56:52 on dist-test-slave-tk9z"
server_key: "5c50efb0199c21a6f23dc9362b5d3c4d"
server_key_iv: "d6f9a4226fa1b9c991966a4f39f8fa60"
server_key_version: "encryptionkey@0"
I20260808 12:56:52.181134 29082 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 12:56:52.184492 29104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:52.185696 29082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 12:56:52.185873 29082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "71d7843a8a424cdf81951041cd817cff"
format_stamp: "Formatted at 2026-08-08 12:56:52 on dist-test-slave-tk9z"
server_key: "5c50efb0199c21a6f23dc9362b5d3c4d"
server_key_iv: "d6f9a4226fa1b9c991966a4f39f8fa60"
server_key_version: "encryptionkey@0"
I20260808 12:56:52.186067 29082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:52.203572 29082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:52.208899 29082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:52.209221 29082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:52.210175 29082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:56:52.211602 29082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:52.211675 29082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:52.211766 29082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:52.211825 29082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:52.235754 28872 heartbeater.cc:503] Master 127.24.167.62:37165 was elected leader, sending a full tablet report...
I20260808 12:56:52.247246 29082 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:44477
I20260808 12:56:52.247294 29258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:44477 every 8 connection(s)
I20260808 12:56:52.248526 29082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:56:52.250185 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 29082
I20260808 12:56:52.250411 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:56:52.251020 25244 external_mini_cluster.cc:1428] Setting key 767ac59a33b60b8cd817e31c01771667
Aug 08 12:56:52 dist-test-slave-tk9z krb5kdc[28547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193812, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:52.271064 29262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:52.251600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:38183 (local address 127.24.167.62:37165)
0808 12:56:52.252355 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:52.252361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:52.253271 (+ 910us) server_negotiation.cc:407] Connection header received
0808 12:56:52.255258 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:52.255263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:52.255368 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:52.255486 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:52.257652 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:52.258824 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:52.260283 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:52.260670 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:52.260761 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:52.264440 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:52.264445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:52.264448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:52.264482 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:52.267652 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:52.268442 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:52.268448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:52.268452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:52.268589 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:52.269071 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:52.269078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:52.269081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:52.269396 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:52.269567 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:52.270351 (+ 784us) server_negotiation.cc:299] Negotiation successful
0808 12:56:52.270628 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":486,"threads_started":1}
I20260808 12:56:52.272058 29264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:52.252227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:37165 (local address 127.24.167.3:38183)
0808 12:56:52.253128 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:52.253166 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:52.255008 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:52.255815 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:52.255826 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:52.256639 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:52.257450 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:52.257479 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:52.258980 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:52.258984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:52.260165 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:52.260177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:52.260858 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:52.260863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:52.261042 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:52.261629 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:52.261650 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:52.264186 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:52.267822 (+ 3636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:52.267832 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:52.267849 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:52.268304 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:52.268728 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:52.268733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:52.268738 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:52.268951 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:52.269539 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:52.269547 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:52.269944 (+ 397us) client_negotiation.cc:769] Sending connection context
0808 12:56:52.270234 (+ 290us) client_negotiation.cc:240] Negotiation successful
0808 12:56:52.270860 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":343,"threads_started":1}
I20260808 12:56:52.273473 29259 heartbeater.cc:347] Connected to a master server at 127.24.167.62:37165
I20260808 12:56:52.273759 29259 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:52.274580 29259 heartbeater.cc:511] Master 127.24.167.62:37165 requested a full tablet report, sending...
I20260808 12:56:52.276063 28603 ts_manager.cc:194] Registered new tserver with Master: 71d7843a8a424cdf81951041cd817cff (127.24.167.3:44477)
I20260808 12:56:52.276952 28603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:38183
I20260808 12:56:52.279333 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 12:56:52.314182 29262 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:56:52.311535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.24.167.62:37165)
0808 12:56:52.311845 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:52.311853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:52.312139 (+ 286us) server_negotiation.cc:407] Connection header received
0808 12:56:52.312350 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:52.312360 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:52.312425 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:52.312534 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:52.313879 (+ 1345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38554: BlockingRecv error: recv got EOF from 127.0.0.1:38554 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
I20260808 12:56:52.317826 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 28687
W20260808 12:56:52.320406 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 28687: Not authorized: Client connection negotiation failed: client connection to 127.24.167.1:43137: server requires authentication, but client does not have Kerberos credentials available
I20260808 12:56:52.320619 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 28687
I20260808 12:56:52.345319 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 28880
W20260808 12:56:52.348215 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 28880: Not authorized: Client connection negotiation failed: client connection to 127.24.167.2:34355: server requires authentication, but client does not have Kerberos credentials available
I20260808 12:56:52.348357 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 28880
I20260808 12:56:52.372871 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 29082
W20260808 12:56:52.375784 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 29082: Not authorized: Client connection negotiation failed: client connection to 127.24.167.3:44477: server requires authentication, but client does not have Kerberos credentials available
I20260808 12:56:52.375967 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 29082
I20260808 12:56:52.400866 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 28564
I20260808 12:56:52.402207 28646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:56:52.505807 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 28564
2026-08-08T12:56:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3880 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:56:52 dist-test-slave-tk9z krb5kdc[29288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:56:52 dist-test-slave-tk9z krb5kdc[29288](info): set up 2 sockets
Aug 08 12:56:52 dist-test-slave-tk9z krb5kdc[29288](info): commencing operation
krb5kdc: starting...
W20260808 12:56:53.785008 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:56:53 dist-test-slave-tk9z krb5kdc[29288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:56:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:56:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 12:56:53.989360 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:38857
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:38857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:54.317996 29308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:54.318347 29308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:54.318571 29308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:54.325837 29308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:56:54.325982 29308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:54.326092 29308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:54.326195 29308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:56:54.326283 29308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:56:54.336171 29308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:38857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:38857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:54.337514 29308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:54.338907 29308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:54.350601 29314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:54.350809 29315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:54.350984 29317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:54.351744 29308 server_base.cc:1034] running on GCE node
I20260808 12:56:54.352524 29308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:54.353847 29308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:54.355058 29308 hybrid_clock.cc:648] HybridClock initialized: now 1786193814354999 us; error 52 us; skew 500 ppm
I20260808 12:56:54.355432 29308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:54 dist-test-slave-tk9z krb5kdc[29288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193814, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:54.363171 29308 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:56:54.364693 29308 webserver.cc:460] Webserver started at http://127.24.167.62:46831/ using document root <none> and password file <none>
I20260808 12:56:54.365397 29308 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:54.365542 29308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:54.365859 29308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:54.369213 29308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "8ea37a688cc049ce9f6339fdc4be599e"
format_stamp: "Formatted at 2026-08-08 12:56:54 on dist-test-slave-tk9z"
server_key: "c2049962acd9781a382f529b1dfa4239"
server_key_iv: "3acb8b4bfd31e31c9f2a40706793d9da"
server_key_version: "encryptionkey@0"
I20260808 12:56:54.370081 29308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "8ea37a688cc049ce9f6339fdc4be599e"
format_stamp: "Formatted at 2026-08-08 12:56:54 on dist-test-slave-tk9z"
server_key: "c2049962acd9781a382f529b1dfa4239"
server_key_iv: "3acb8b4bfd31e31c9f2a40706793d9da"
server_key_version: "encryptionkey@0"
I20260808 12:56:54.375512 29308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260808 12:56:54.379158 29328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:54.380506 29308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 12:56:54.380705 29308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "8ea37a688cc049ce9f6339fdc4be599e"
format_stamp: "Formatted at 2026-08-08 12:56:54 on dist-test-slave-tk9z"
server_key: "c2049962acd9781a382f529b1dfa4239"
server_key_iv: "3acb8b4bfd31e31c9f2a40706793d9da"
server_key_version: "encryptionkey@0"
I20260808 12:56:54.380919 29308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:54.398012 29308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:54.402694 29308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:54.403018 29308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:54.421818 29308 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:38857
I20260808 12:56:54.421854 29400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:38857 every 8 connection(s)
I20260808 12:56:54.423278 29308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:56:54.427137 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:54.428519 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 29308
I20260808 12:56:54.428773 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:56:54.429425 25244 external_mini_cluster.cc:1428] Setting key e82eb34886f35230120578b137d06813
I20260808 12:56:54.441527 29404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e: Bootstrap starting.
Aug 08 12:56:54 dist-test-slave-tk9z krb5kdc[29288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:54.444521 29404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:54.445658 29404 log.cc:824] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e: Log is configured to *not* fsync() on all Append() calls
I20260808 12:56:54.448612 29404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e: No bootstrap required, opened a new log
I20260808 12:56:54.452047 29408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:54.431365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.24.167.62:38857)
0808 12:56:54.432234 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:54.432248 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:54.432308 (+ 60us) server_negotiation.cc:407] Connection header received
0808 12:56:54.434014 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:54.434042 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:54.434589 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:54.435021 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:54.436340 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:54.438453 (+ 2113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:54.439736 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:54.440777 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:54.440953 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:54.444853 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:54.444861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:54.444878 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:54.444918 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:54.448054 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:54.448727 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:54.448741 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:54.448747 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:54.448908 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:54.449283 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:54.449288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:54.449291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:54.449823 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:54.450139 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:54.450667 (+ 528us) server_negotiation.cc:299] Negotiation successful
0808 12:56:54.451074 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":354,"threads_started":1}
I20260808 12:56:54.455783 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 12:56:54.456889 29404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea37a688cc049ce9f6339fdc4be599e" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 38857 } }
I20260808 12:56:54.457259 29404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:54.457360 29404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea37a688cc049ce9f6339fdc4be599e, State: Initialized, Role: FOLLOWER
I20260808 12:56:54.457902 29404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea37a688cc049ce9f6339fdc4be599e" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 38857 } }
I20260808 12:56:54.458149 29404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:56:54.458289 29404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:56:54.458482 29404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:54.460645 29404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea37a688cc049ce9f6339fdc4be599e" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 38857 } }
I20260808 12:56:54.461149 29404 leader_election.cc:302] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ea37a688cc049ce9f6339fdc4be599e; no voters:
I20260808 12:56:54.461824 29404 leader_election.cc:288] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:56:54.461921 29410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:54.463611 29410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [term 1 LEADER]: Becoming Leader. State: Replica: 8ea37a688cc049ce9f6339fdc4be599e, State: Running, Role: LEADER
I20260808 12:56:54.464217 29410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea37a688cc049ce9f6339fdc4be599e" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 38857 } }
I20260808 12:56:54.464450 29404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:56:54.467723 29413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ea37a688cc049ce9f6339fdc4be599e. Latest consensus state: current_term: 1 leader_uuid: "8ea37a688cc049ce9f6339fdc4be599e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea37a688cc049ce9f6339fdc4be599e" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 38857 } } }
I20260808 12:56:54.468595 29413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [sys.catalog]: This master's current role is: LEADER
I20260808 12:56:54.468286 29412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ea37a688cc049ce9f6339fdc4be599e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea37a688cc049ce9f6339fdc4be599e" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 38857 } } }
I20260808 12:56:54.469058 29412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e [sys.catalog]: This master's current role is: LEADER
I20260808 12:56:54.478961 29421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:56:54.484833 29421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:56:54.494019 29421 catalog_manager.cc:1347] Generated new cluster ID: 603cfccd8c6e44638097bbd8dc9353e6
I20260808 12:56:54.494140 29421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:56:54.505605 29421 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:56:54.506973 29421 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:56:54.514405 29421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ea37a688cc049ce9f6339fdc4be599e: Generated new TSK 0
I20260808 12:56:54.515475 29421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 12:56:54.520393 29408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:54.510961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.24.167.62:38857)
0808 12:56:54.511140 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:54.511151 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:54.512743 (+ 1592us) server_negotiation.cc:407] Connection header received
0808 12:56:54.512949 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:54.512956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:54.513046 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:54.513167 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:54.514508 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:54.515285 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:54.517001 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:54.517558 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:54.517724 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:54.518976 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:54.518981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:54.518984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:56:54.519597 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:54.519659 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:56:54.519686 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:54.519816 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:54.519972 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 12:56:54.520072 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 12:56:54.527245 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 29308
I20260808 12:56:54.528910 29391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:56:54.619340 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 29308
2026-08-08T12:56:54Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2106 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:56:54 dist-test-slave-tk9z krb5kdc[29450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:56:54 dist-test-slave-tk9z krb5kdc[29450](info): set up 2 sockets
Aug 08 12:56:54 dist-test-slave-tk9z krb5kdc[29450](info): commencing operation
krb5kdc: starting...
W20260808 12:56:55.924409 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.264s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:56:56 dist-test-slave-tk9z krb5kdc[29450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:56:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:56:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 12:56:56.139369 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:41031
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:41031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:56.475174 29464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:56.475564 29464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:56.475725 29464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:56.483114 29464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:56:56.483243 29464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:56.483345 29464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:56.483457 29464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:56:56.483534 29464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:56:56.493350 29464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:41031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:41031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:56.494720 29464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:56.496043 29464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:56.507872 29469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:56.507839 29472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:56.508625 29464 server_base.cc:1034] running on GCE node
W20260808 12:56:56.508816 29470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:56.509663 29464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:56.511123 29464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:56.512346 29464 hybrid_clock.cc:648] HybridClock initialized: now 1786193816512293 us; error 62 us; skew 500 ppm
I20260808 12:56:56.512689 29464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:56 dist-test-slave-tk9z krb5kdc[29450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193816, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:56.515872 29464 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:56:56.517328 29464 webserver.cc:460] Webserver started at http://127.24.167.62:36051/ using document root <none> and password file <none>
I20260808 12:56:56.518014 29464 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:56.518158 29464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:56.518515 29464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:56.521540 29464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "20985878d95e4208b6bf70ecfbf15aac"
format_stamp: "Formatted at 2026-08-08 12:56:56 on dist-test-slave-tk9z"
server_key: "c58aa2b6d6eb513d192e3af744a19962"
server_key_iv: "7e23c83f5c92513965a270d497e7997a"
server_key_version: "encryptionkey@0"
I20260808 12:56:56.522308 29464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "20985878d95e4208b6bf70ecfbf15aac"
format_stamp: "Formatted at 2026-08-08 12:56:56 on dist-test-slave-tk9z"
server_key: "c58aa2b6d6eb513d192e3af744a19962"
server_key_iv: "7e23c83f5c92513965a270d497e7997a"
server_key_version: "encryptionkey@0"
I20260808 12:56:56.527408 29464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 12:56:56.530786 29480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:56.532090 29464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 12:56:56.532301 29464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "20985878d95e4208b6bf70ecfbf15aac"
format_stamp: "Formatted at 2026-08-08 12:56:56 on dist-test-slave-tk9z"
server_key: "c58aa2b6d6eb513d192e3af744a19962"
server_key_iv: "7e23c83f5c92513965a270d497e7997a"
server_key_version: "encryptionkey@0"
I20260808 12:56:56.532503 29464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:56.576045 29464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:56.581122 29464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:56.581466 29464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:56.601665 29464 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:41031
I20260808 12:56:56.601688 29547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:41031 every 8 connection(s)
I20260808 12:56:56.603168 29464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:56:56.606988 29548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:56:56.610783 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 29464
I20260808 12:56:56.611040 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:56:56.611690 25244 external_mini_cluster.cc:1428] Setting key efa0889cfcc17b17330410dd6e8bb348
I20260808 12:56:56.618125 29548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac: Bootstrap starting.
I20260808 12:56:56.621773 29548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac: Neither blocks nor log segments found. Creating new log.
I20260808 12:56:56.623023 29548 log.cc:824] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac: Log is configured to *not* fsync() on all Append() calls
I20260808 12:56:56.626519 29548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac: No bootstrap required, opened a new log
Aug 08 12:56:56 dist-test-slave-tk9z krb5kdc[29450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:56.635440 29548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20985878d95e4208b6bf70ecfbf15aac" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41031 } }
I20260808 12:56:56.635978 29548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:56:56.636119 29548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20985878d95e4208b6bf70ecfbf15aac, State: Initialized, Role: FOLLOWER
I20260808 12:56:56.636158 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:56.613795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55666 (local address 127.24.167.62:41031)
0808 12:56:56.614722 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:56.614733 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:56.614778 (+ 45us) server_negotiation.cc:407] Connection header received
0808 12:56:56.616530 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:56.616554 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:56.616939 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:56.617247 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:56.619074 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:56.620403 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:56.622205 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:56.622919 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:56.623070 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:56.628124 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:56.628133 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:56.628147 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:56.628193 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:56.631851 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:56.632648 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:56.632655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:56.632661 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:56.632835 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:56.633232 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:56.633241 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:56.633248 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:56.633820 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:56.634152 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:56.634615 (+ 463us) server_negotiation.cc:299] Negotiation successful
0808 12:56:56.635026 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":480,"threads_started":1}
I20260808 12:56:56.636833 29548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20985878d95e4208b6bf70ecfbf15aac" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41031 } }
I20260808 12:56:56.637156 29548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:56:56.637321 29548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:56:56.637530 29548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:56:56.640828 29548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20985878d95e4208b6bf70ecfbf15aac" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41031 } }
I20260808 12:56:56.641489 29548 leader_election.cc:302] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20985878d95e4208b6bf70ecfbf15aac; no voters:
I20260808 12:56:56.642132 29548 leader_election.cc:288] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:56:56.642204 29558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:56:56.644800 29548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:56:56.645227 29558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [term 1 LEADER]: Becoming Leader. State: Replica: 20985878d95e4208b6bf70ecfbf15aac, State: Running, Role: LEADER
I20260808 12:56:56.645814 29558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20985878d95e4208b6bf70ecfbf15aac" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41031 } }
I20260808 12:56:56.650776 29560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20985878d95e4208b6bf70ecfbf15aac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20985878d95e4208b6bf70ecfbf15aac" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41031 } } }
I20260808 12:56:56.650797 29561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20985878d95e4208b6bf70ecfbf15aac. Latest consensus state: current_term: 1 leader_uuid: "20985878d95e4208b6bf70ecfbf15aac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20985878d95e4208b6bf70ecfbf15aac" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41031 } } }
I20260808 12:56:56.651237 29560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [sys.catalog]: This master's current role is: LEADER
I20260808 12:56:56.651247 29561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac [sys.catalog]: This master's current role is: LEADER
I20260808 12:56:56.654052 29577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:56:56.658710 29577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:56:56.666898 29577 catalog_manager.cc:1347] Generated new cluster ID: 3923624c951d43719fc30b4fea7e0efe
I20260808 12:56:56.667021 29577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:56:56.674225 29577 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:56:56.676357 29577 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:56:56.697338 29577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20985878d95e4208b6bf70ecfbf15aac: Generated new TSK 0
I20260808 12:56:56.698547 29577 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 12:56:56.755517 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:41031
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:57.117806 29591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:57.118091 29591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:57.118271 29591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:57.125648 29591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:57.125759 29591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:57.125939 29591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:56:57.135869 29591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:41031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:57.137472 29591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:57.138952 29591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:57.152175 29600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:57.152130 29604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:57.152226 29602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:57.155972 29591 server_base.cc:1034] running on GCE node
I20260808 12:56:57.156672 29591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:57.157634 29591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:57.158859 29591 hybrid_clock.cc:648] HybridClock initialized: now 1786193817158795 us; error 57 us; skew 500 ppm
I20260808 12:56:57.159185 29591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:57 dist-test-slave-tk9z krb5kdc[29450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193817, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:57.162735 29591 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 12:56:57.164394 29591 webserver.cc:460] Webserver started at http://127.24.167.1:45145/ using document root <none> and password file <none>
I20260808 12:56:57.165119 29591 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:57.165261 29591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:57.165575 29591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:57.168537 29591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "a64b001fc8d24655890cf5de9e43dfe5"
format_stamp: "Formatted at 2026-08-08 12:56:57 on dist-test-slave-tk9z"
server_key: "0495d79bb57302e3437846bf343a7904"
server_key_iv: "bd534f8fa95ec3a7b9dda73747fc1553"
server_key_version: "encryptionkey@0"
I20260808 12:56:57.169317 29591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "a64b001fc8d24655890cf5de9e43dfe5"
format_stamp: "Formatted at 2026-08-08 12:56:57 on dist-test-slave-tk9z"
server_key: "0495d79bb57302e3437846bf343a7904"
server_key_iv: "bd534f8fa95ec3a7b9dda73747fc1553"
server_key_version: "encryptionkey@0"
I20260808 12:56:57.174578 29591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 12:56:57.178143 29616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:57.179656 29591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:56:57.179843 29591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "a64b001fc8d24655890cf5de9e43dfe5"
format_stamp: "Formatted at 2026-08-08 12:56:57 on dist-test-slave-tk9z"
server_key: "0495d79bb57302e3437846bf343a7904"
server_key_iv: "bd534f8fa95ec3a7b9dda73747fc1553"
server_key_version: "encryptionkey@0"
I20260808 12:56:57.180058 29591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:57.199518 29591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:57.204064 29591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:57.204397 29591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:57.205729 29591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:56:57.207274 29591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:57.207387 29591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:57.207535 29591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:57.207607 29591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:57.242475 29591 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:42537
I20260808 12:56:57.242533 29785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:42537 every 8 connection(s)
I20260808 12:56:57.243789 29591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:56:57.249935 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 29591
I20260808 12:56:57.250205 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:56:57.251133 25244 external_mini_cluster.cc:1428] Setting key 2ebffdb19f5928c969526c951e10532e
Aug 08 12:56:57 dist-test-slave-tk9z krb5kdc[29450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193817, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:57.269351 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:57.246532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:50187 (local address 127.24.167.62:41031)
0808 12:56:57.246997 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:57.247002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:57.247853 (+ 851us) server_negotiation.cc:407] Connection header received
0808 12:56:57.250004 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:57.250009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:57.250122 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:57.250217 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:57.252385 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:57.253531 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:57.256235 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:57.256660 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:57.256771 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:57.261702 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:57.261711 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:57.261719 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:57.261768 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:57.265594 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:57.266474 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:57.266482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:57.266488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:57.266652 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:57.267132 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:57.267140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:57.267146 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:57.267575 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:57.267737 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:57.268568 (+ 831us) server_negotiation.cc:299] Negotiation successful
0808 12:56:57.268851 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":260,"threads_started":1}
I20260808 12:56:57.270025 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:57.246976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41031 (local address 127.24.167.1:50187)
0808 12:56:57.247701 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:57.247740 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:57.249650 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:57.250545 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:57.250559 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:57.251378 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:57.252164 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:57.252190 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:57.253698 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:57.253703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:57.256047 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:57.256067 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:57.256920 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:57.256930 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:57.257304 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:57.258083 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:57.258125 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:57.261416 (+ 3291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:57.265771 (+ 4355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:57.265779 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:57.265800 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:57.266293 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:57.266780 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:57.266784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:57.266786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:57.266990 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:57.267714 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:57.267720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:57.268118 (+ 398us) client_negotiation.cc:769] Sending connection context
0808 12:56:57.268362 (+ 244us) client_negotiation.cc:240] Negotiation successful
0808 12:56:57.268803 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":304,"threads_started":1}
I20260808 12:56:57.271747 29790 heartbeater.cc:347] Connected to a master server at 127.24.167.62:41031
I20260808 12:56:57.272131 29790 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:57.273195 29790 heartbeater.cc:511] Master 127.24.167.62:41031 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
I20260808 12:56:57.276352 29503 ts_manager.cc:194] Registered new tserver with Master: a64b001fc8d24655890cf5de9e43dfe5 (127.24.167.1:42537)
I20260808 12:56:57.278228 29503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:50187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 12:56:57.305596 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:41031
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:57.665416 29798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:57.665702 29798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:57.665880 29798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:57.673087 29798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:57.673203 29798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:57.673410 29798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:56:57.683655 29798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:41031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:57.685040 29798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:57.686365 29798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:57.699050 29803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:57.699110 29807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:57.699013 29804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:57.700883 29798 server_base.cc:1034] running on GCE node
I20260808 12:56:57.701499 29798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:57.702448 29798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:57.703699 29798 hybrid_clock.cc:648] HybridClock initialized: now 1786193817703629 us; error 56 us; skew 500 ppm
I20260808 12:56:57.704048 29798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:57 dist-test-slave-tk9z krb5kdc[29450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193817, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:57.707234 29798 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 12:56:57.708611 29798 webserver.cc:460] Webserver started at http://127.24.167.2:38199/ using document root <none> and password file <none>
I20260808 12:56:57.709340 29798 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:57.709481 29798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:57.709779 29798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:57.712800 29798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "6700f375bfed49358b0196e27ec76a60"
format_stamp: "Formatted at 2026-08-08 12:56:57 on dist-test-slave-tk9z"
server_key: "2317a976ff06925a441eda77db73fb67"
server_key_iv: "c6d1ba8b43114d679fde6affb5a7f890"
server_key_version: "encryptionkey@0"
I20260808 12:56:57.713626 29798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "6700f375bfed49358b0196e27ec76a60"
format_stamp: "Formatted at 2026-08-08 12:56:57 on dist-test-slave-tk9z"
server_key: "2317a976ff06925a441eda77db73fb67"
server_key_iv: "c6d1ba8b43114d679fde6affb5a7f890"
server_key_version: "encryptionkey@0"
I20260808 12:56:57.719254 29798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 12:56:57.722944 29817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:57.724381 29798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 12:56:57.724563 29798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "6700f375bfed49358b0196e27ec76a60"
format_stamp: "Formatted at 2026-08-08 12:56:57 on dist-test-slave-tk9z"
server_key: "2317a976ff06925a441eda77db73fb67"
server_key_iv: "c6d1ba8b43114d679fde6affb5a7f890"
server_key_version: "encryptionkey@0"
I20260808 12:56:57.724783 29798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:57.742690 29798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:57.746960 29798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:57.747314 29798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:57.748427 29798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:56:57.749862 29798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:57.749962 29798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:57.750092 29798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:57.750166 29798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:57.786306 29798 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:42005
I20260808 12:56:57.786361 29984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:42005 every 8 connection(s)
I20260808 12:56:57.787667 29798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:56:57.790091 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 29798
I20260808 12:56:57.790325 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:56:57.790964 25244 external_mini_cluster.cc:1428] Setting key 093d835cd52cb8706e34f05df159d14d
Aug 08 12:56:57 dist-test-slave-tk9z krb5kdc[29450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193817, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:57.812350 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:57.791968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:44409 (local address 127.24.167.62:41031)
0808 12:56:57.792439 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:57.792444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:57.794195 (+ 1751us) server_negotiation.cc:407] Connection header received
0808 12:56:57.796095 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:57.796101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:57.796194 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:57.796308 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:57.797831 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:57.798627 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:57.800209 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:57.800632 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:57.800762 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:57.805142 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:57.805147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:57.805150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:57.805185 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:57.808426 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:57.809500 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:57.809505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:57.809507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:57.809654 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:57.810241 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:57.810245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:57.810247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:57.810748 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:57.810895 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:57.811864 (+ 969us) server_negotiation.cc:299] Negotiation successful
0808 12:56:57.812058 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":241,"threads_started":1}
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
I20260808 12:56:57.813925 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:57.791210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41031 (local address 127.24.167.2:44409)
0808 12:56:57.794036 (+ 2826us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:57.794080 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:57.795821 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:57.796558 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:57.796566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:57.797087 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:57.797654 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:57.797672 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:57.798782 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:57.798792 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:57.800048 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:57.800070 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:57.800873 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:57.800880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:57.801116 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:57.801805 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:57.801834 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:57.804865 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:57.808596 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:57.808609 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:57.808630 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:57.809350 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:57.809785 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:57.809794 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:57.809809 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:57.810094 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:57.810908 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:57.810920 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:57.811417 (+ 497us) client_negotiation.cc:769] Sending connection context
0808 12:56:57.811759 (+ 342us) client_negotiation.cc:240] Negotiation successful
0808 12:56:57.812390 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2562,"thread_start_us":2286,"threads_started":1}
I20260808 12:56:57.815605 29987 heartbeater.cc:347] Connected to a master server at 127.24.167.62:41031
I20260808 12:56:57.816105 29987 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
I20260808 12:56:57.817168 29987 heartbeater.cc:511] Master 127.24.167.62:41031 requested a full tablet report, sending...
I20260808 12:56:57.818974 29503 ts_manager.cc:194] Registered new tserver with Master: 6700f375bfed49358b0196e27ec76a60 (127.24.167.2:42005)
I20260808 12:56:57.820246 29503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:44409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 12:56:57.843083 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:41031
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:56:58.184661 29996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:56:58.184953 29996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:56:58.185101 29996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:56:58.192365 29996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:56:58.192502 29996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:56:58.192695 29996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:56:58.203238 29996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:41031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:56:58.204553 29996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:56:58.205852 29996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:56:58.219533 30003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:58.219493 30007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:56:58.219508 30004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:56:58.220880 29996 server_base.cc:1034] running on GCE node
I20260808 12:56:58.221517 29996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:56:58.222580 29996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:56:58.223815 29996 hybrid_clock.cc:648] HybridClock initialized: now 1786193818223742 us; error 61 us; skew 500 ppm
I20260808 12:56:58.224135 29996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:56:58 dist-test-slave-tk9z krb5kdc[29450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193818, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:56:58.227684 29996 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 12:56:58.229382 29996 webserver.cc:460] Webserver started at http://127.24.167.3:39441/ using document root <none> and password file <none>
I20260808 12:56:58.230098 29996 fs_manager.cc:359] Metadata directory not provided
I20260808 12:56:58.230242 29996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:56:58.230625 29996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:56:58.233702 29996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "4f6c6c5dc6e24ace9f3c4561d9aefdd2"
format_stamp: "Formatted at 2026-08-08 12:56:58 on dist-test-slave-tk9z"
server_key: "1feaff8743dd17c45193d620daacb1a2"
server_key_iv: "809c0b58a526b09ebb8e7890fc340cd2"
server_key_version: "encryptionkey@0"
I20260808 12:56:58.234633 29996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "4f6c6c5dc6e24ace9f3c4561d9aefdd2"
format_stamp: "Formatted at 2026-08-08 12:56:58 on dist-test-slave-tk9z"
server_key: "1feaff8743dd17c45193d620daacb1a2"
server_key_iv: "809c0b58a526b09ebb8e7890fc340cd2"
server_key_version: "encryptionkey@0"
I20260808 12:56:58.239923 29996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 12:56:58.243741 30015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:58.245381 29996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 12:56:58.245589 29996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "4f6c6c5dc6e24ace9f3c4561d9aefdd2"
format_stamp: "Formatted at 2026-08-08 12:56:58 on dist-test-slave-tk9z"
server_key: "1feaff8743dd17c45193d620daacb1a2"
server_key_iv: "809c0b58a526b09ebb8e7890fc340cd2"
server_key_version: "encryptionkey@0"
I20260808 12:56:58.245939 29996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:56:58.266185 29996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:56:58.271203 29996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:56:58.271584 29996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:56:58.272661 29996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:56:58.274184 29996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:56:58.274305 29996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:58.274497 29996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:56:58.274582 29996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:56:58.281487 29790 heartbeater.cc:503] Master 127.24.167.62:41031 was elected leader, sending a full tablet report...
I20260808 12:56:58.312582 29996 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:33027
I20260808 12:56:58.312623 30185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:33027 every 8 connection(s)
I20260808 12:56:58.314154 29996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:56:58.324481 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 29996
I20260808 12:56:58.324730 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:56:58.325248 25244 external_mini_cluster.cc:1428] Setting key 35c0d5ad69f73dee7bb9fc0af0869b88
Aug 08 12:56:58 dist-test-slave-tk9z krb5kdc[29450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193818, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:58.334183 30189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:58.316826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:57303 (local address 127.24.167.62:41031)
0808 12:56:58.317278 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:58.317283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:58.318408 (+ 1125us) server_negotiation.cc:407] Connection header received
0808 12:56:58.320269 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:58.320276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:58.320367 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:58.320502 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:58.321954 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:58.322801 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:58.324362 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:58.324843 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:58.324992 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:58.328811 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:58.328816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:58.328819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:58.328850 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:58.331557 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:58.332219 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:58.332223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:58.332225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:58.332341 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:58.332746 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:58.332749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:58.332751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:58.333008 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:58.333117 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:58.333764 (+ 647us) server_negotiation.cc:299] Negotiation successful
0808 12:56:58.333922 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":218,"threads_started":1}
I20260808 12:56:58.335192 30190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:58.317469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41031 (local address 127.24.167.3:57303)
0808 12:56:58.318203 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0808 12:56:58.318241 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 12:56:58.319984 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:56:58.320729 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:56:58.320738 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:56:58.321239 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:56:58.321781 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:58.321797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:58.322969 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:58.322973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:58.324202 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:56:58.324213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:56:58.325101 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:56:58.325110 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:56:58.325334 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:58.326071 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:56:58.326099 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:56:58.328614 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:56:58.331679 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:58.331686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:58.331698 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:58.332116 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:58.332438 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:56:58.332441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:56:58.332466 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:56:58.332647 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:56:58.333114 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:56:58.333120 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:56:58.333423 (+ 303us) client_negotiation.cc:769] Sending connection context
0808 12:56:58.333678 (+ 255us) client_negotiation.cc:240] Negotiation successful
0808 12:56:58.334049 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":301,"threads_started":1}
I20260808 12:56:58.336612 30186 heartbeater.cc:347] Connected to a master server at 127.24.167.62:41031
I20260808 12:56:58.336975 30186 heartbeater.cc:464] Registering TS with master...
I20260808 12:56:58.337793 30186 heartbeater.cc:511] Master 127.24.167.62:41031 requested a full tablet report, sending...
I20260808 12:56:58.339304 29503 ts_manager.cc:194] Registered new tserver with Master: 4f6c6c5dc6e24ace9f3c4561d9aefdd2 (127.24.167.3:33027)
I20260808 12:56:58.340269 29503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:57303
I20260808 12:56:58.341120 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 12:56:58 dist-test-slave-tk9z krb5kdc[29450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193818, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 12:56:58 dist-test-slave-tk9z krb5kdc[29450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193818, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:56:58.396528 30189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:58.381402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.24.167.62:41031)
0808 12:56:58.381685 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:58.381694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:58.382073 (+ 379us) server_negotiation.cc:407] Connection header received
0808 12:56:58.382311 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:58.382315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:58.382433 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:58.382554 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:58.384095 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:58.384918 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:58.386196 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:58.386716 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:58.386863 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:58.391077 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:58.391082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:58.391086 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:58.391127 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:58.393685 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:58.394366 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:58.394399 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:58.394405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:58.394549 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:58.394992 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:58.394997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:58.395000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:58.395356 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:58.395621 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:58.395928 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 12:56:58.396189 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260808 12:56:58.397197 29502 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55670
I20260808 12:56:58.399194 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 29591
Aug 08 12:56:58 dist-test-slave-tk9z krb5kdc[29450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193818, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:56:58.415577 30200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:58.400411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.24.167.1:42537)
0808 12:56:58.400990 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:58.400997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:58.401024 (+ 27us) server_negotiation.cc:407] Connection header received
0808 12:56:58.401080 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:58.401086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:58.401287 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:58.401408 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:58.402717 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:58.403719 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:58.404994 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:58.405586 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:58.405687 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:58.409632 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:58.409645 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:58.409659 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:58.409694 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:58.412695 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:58.413555 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:58.413562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:58.413564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:58.413687 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:58.414046 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:58.414049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:58.414051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:58.414473 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:58.414713 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:58.414943 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 12:56:58.415135 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":289,"threads_started":1}
W20260808 12:56:58.416842 29775 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52362
W20260808 12:56:58.418004 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 29591: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 12:56:58.418180 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 29591
I20260808 12:56:58.441196 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 29798
Aug 08 12:56:58 dist-test-slave-tk9z krb5kdc[29450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193818, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 12:56:58.457194 30201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:58.442154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.24.167.2:42005)
0808 12:56:58.442786 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:58.442793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:58.442840 (+ 47us) server_negotiation.cc:407] Connection header received
0808 12:56:58.442912 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:58.442919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:58.443119 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:58.443247 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:58.444526 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:58.445602 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:58.446979 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:58.447563 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:58.447658 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:58.451759 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:58.451769 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:58.451783 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:58.451831 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:58.454742 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:58.455329 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:58.455336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:58.455338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:58.455455 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:58.455771 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:58.455774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:58.455776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:58.456125 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:58.456360 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:58.456606 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 12:56:58.456832 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":363,"threads_started":1}
W20260808 12:56:58.458606 29979 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58710
W20260808 12:56:58.459698 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 29798: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 12:56:58.459865 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 29798
I20260808 12:56:58.483687 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 29996
Aug 08 12:56:58 dist-test-slave-tk9z krb5kdc[29450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193818, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 12:56:58.500958 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:56:58.484803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.24.167.3:33027)
0808 12:56:58.485496 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0808 12:56:58.485503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:56:58.485534 (+ 31us) server_negotiation.cc:407] Connection header received
0808 12:56:58.485604 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:56:58.485611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:56:58.485823 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:56:58.486031 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:56:58.487491 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:58.488576 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:58.489858 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:56:58.490622 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:56:58.490772 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:56:58.495250 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:56:58.495261 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:56:58.495278 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:56:58.495317 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:56:58.498238 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:58.498980 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:58.498988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:58.498990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:58.499126 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:56:58.499481 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:56:58.499485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:56:58.499487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:56:58.499864 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:56:58.500119 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0808 12:56:58.500394 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 12:56:58.500641 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":359,"threads_started":1}
W20260808 12:56:58.502341 30177 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51370
W20260808 12:56:58.503566 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 29996: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 12:56:58.503732 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 29996
I20260808 12:56:58.528214 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 29464
I20260808 12:56:58.529515 29542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:56:58.633944 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 29464
2026-08-08T12:56:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4022 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:56:58 dist-test-slave-tk9z krb5kdc[30209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:56:58 dist-test-slave-tk9z krb5kdc[30209](info): set up 2 sockets
Aug 08 12:56:58 dist-test-slave-tk9z krb5kdc[30209](info): commencing operation
krb5kdc: starting...
W20260808 12:56:59.941347 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:57:00 dist-test-slave-tk9z krb5kdc[30209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:57:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:57:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 12:57:00.169662 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:39839
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:39839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:00.503394 30227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:00.503702 30227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:00.503854 30227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:00.511091 30227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:57:00.511216 30227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:00.511302 30227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:00.511404 30227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:57:00.511488 30227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:57:00.521320 30227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:39839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:39839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:00.522703 30227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:00.524122 30227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:00.536470 30236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:00.536521 30238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:00.536497 30240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:00.537497 30227 server_base.cc:1034] running on GCE node
I20260808 12:57:00.538314 30227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:00.539803 30227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:00.541041 30227 hybrid_clock.cc:648] HybridClock initialized: now 1786193820540998 us; error 68 us; skew 500 ppm
I20260808 12:57:00.541389 30227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:00 dist-test-slave-tk9z krb5kdc[30209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193820, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:00.549161 30227 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:57:00.550861 30227 webserver.cc:460] Webserver started at http://127.24.167.62:36879/ using document root <none> and password file <none>
I20260808 12:57:00.551589 30227 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:00.551734 30227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:00.552060 30227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:00.555142 30227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "ad3742a9ef2640c089a05a60456f655f"
format_stamp: "Formatted at 2026-08-08 12:57:00 on dist-test-slave-tk9z"
server_key: "5fbd0f4b838758d0369a3af26cf39b0b"
server_key_iv: "0a50e4567160e6705efa96c4b768a30d"
server_key_version: "encryptionkey@0"
I20260808 12:57:00.555948 30227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "ad3742a9ef2640c089a05a60456f655f"
format_stamp: "Formatted at 2026-08-08 12:57:00 on dist-test-slave-tk9z"
server_key: "5fbd0f4b838758d0369a3af26cf39b0b"
server_key_iv: "0a50e4567160e6705efa96c4b768a30d"
server_key_version: "encryptionkey@0"
I20260808 12:57:00.561231 30227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 12:57:00.564687 30253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:00.566025 30227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 12:57:00.566202 30227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "ad3742a9ef2640c089a05a60456f655f"
format_stamp: "Formatted at 2026-08-08 12:57:00 on dist-test-slave-tk9z"
server_key: "5fbd0f4b838758d0369a3af26cf39b0b"
server_key_iv: "0a50e4567160e6705efa96c4b768a30d"
server_key_version: "encryptionkey@0"
I20260808 12:57:00.566435 30227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:00.582557 30227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:00.587204 30227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:00.587527 30227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:00.606315 30227 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:39839
I20260808 12:57:00.606334 30328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:39839 every 8 connection(s)
I20260808 12:57:00.607779 30227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:57:00.611085 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 30227
I20260808 12:57:00.611299 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:57:00.612130 30329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:00.612229 25244 external_mini_cluster.cc:1428] Setting key 75972561a9ad72fa1cb010d846d9b121
I20260808 12:57:00.627722 30329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f: Bootstrap starting.
Aug 08 12:57:00 dist-test-slave-tk9z krb5kdc[30209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:00.631865 30329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:00.633242 30329 log.cc:824] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:00.636821 30329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f: No bootstrap required, opened a new log
I20260808 12:57:00.638046 30336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:00.614591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.24.167.62:39839)
0808 12:57:00.615545 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:00.615558 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:00.615604 (+ 46us) server_negotiation.cc:407] Connection header received
0808 12:57:00.617396 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:00.617421 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:00.617867 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:00.618236 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:00.620127 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:00.621457 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:00.623296 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:00.624029 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:00.624180 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:00.629059 (+ 4879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:00.629070 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:00.629092 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:00.629145 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:00.633242 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:00.634078 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:00.634084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:00.634091 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:00.634264 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:00.634818 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:00.634826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:00.634833 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:00.635565 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:00.635879 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:00.636262 (+ 383us) server_negotiation.cc:299] Negotiation successful
0808 12:57:00.636701 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":435,"threads_started":1}
I20260808 12:57:00.645437 30329 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3742a9ef2640c089a05a60456f655f" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39839 } }
I20260808 12:57:00.645849 30329 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:00.645979 30329 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3742a9ef2640c089a05a60456f655f, State: Initialized, Role: FOLLOWER
I20260808 12:57:00.646642 30329 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3742a9ef2640c089a05a60456f655f" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39839 } }
I20260808 12:57:00.646890 30329 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:57:00.647063 30329 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:57:00.647222 30329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:00.649399 30329 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3742a9ef2640c089a05a60456f655f" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39839 } }
I20260808 12:57:00.649947 30329 leader_election.cc:302] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad3742a9ef2640c089a05a60456f655f; no voters:
I20260808 12:57:00.650671 30329 leader_election.cc:288] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:57:00.650851 30340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:00.651731 30340 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [term 1 LEADER]: Becoming Leader. State: Replica: ad3742a9ef2640c089a05a60456f655f, State: Running, Role: LEADER
I20260808 12:57:00.653335 30329 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:57:00.653740 30340 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3742a9ef2640c089a05a60456f655f" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39839 } }
I20260808 12:57:00.657087 30341 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad3742a9ef2640c089a05a60456f655f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3742a9ef2640c089a05a60456f655f" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39839 } } }
I20260808 12:57:00.657480 30341 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:00.658475 30350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:57:00.658288 30342 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad3742a9ef2640c089a05a60456f655f. Latest consensus state: current_term: 1 leader_uuid: "ad3742a9ef2640c089a05a60456f655f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3742a9ef2640c089a05a60456f655f" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39839 } } }
I20260808 12:57:00.658658 30342 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:00.664575 30350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:57:00.674254 30350 catalog_manager.cc:1347] Generated new cluster ID: f1ec6078cd014d54b60c8548ed945d4a
I20260808 12:57:00.674434 30350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:57:00.700750 30350 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:57:00.702287 30350 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:57:00.713874 30350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad3742a9ef2640c089a05a60456f655f: Generated new TSK 0
I20260808 12:57:00.715085 30350 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 12:57:00.781957 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:39839
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:01.141112 30372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:01.141415 30372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:01.141613 30372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:01.149107 30372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:01.149225 30372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:01.149444 30372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:57:01.160034 30372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:39839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:01.161372 30372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:01.162803 30372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:01.175931 30377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:01.175854 30378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:01.175892 30381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:01.176829 30372 server_base.cc:1034] running on GCE node
I20260808 12:57:01.177467 30372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:01.178490 30372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:01.179710 30372 hybrid_clock.cc:648] HybridClock initialized: now 1786193821179655 us; error 33 us; skew 500 ppm
I20260808 12:57:01.180120 30372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:01 dist-test-slave-tk9z krb5kdc[30209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193821, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:01.183507 30372 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 12:57:01.184964 30372 webserver.cc:460] Webserver started at http://127.24.167.1:33863/ using document root <none> and password file <none>
I20260808 12:57:01.185623 30372 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:01.185724 30372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:01.185986 30372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:01.188952 30372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "5bac1e472735461086b47098b576e470"
format_stamp: "Formatted at 2026-08-08 12:57:01 on dist-test-slave-tk9z"
server_key: "2f8523a8852f542d938736c5db090991"
server_key_iv: "b5ad8b1cc59e7baab10a155515f76fe9"
server_key_version: "encryptionkey@0"
I20260808 12:57:01.189703 30372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "5bac1e472735461086b47098b576e470"
format_stamp: "Formatted at 2026-08-08 12:57:01 on dist-test-slave-tk9z"
server_key: "2f8523a8852f542d938736c5db090991"
server_key_iv: "b5ad8b1cc59e7baab10a155515f76fe9"
server_key_version: "encryptionkey@0"
I20260808 12:57:01.194832 30372 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 12:57:01.198220 30395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:01.199576 30372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:57:01.199781 30372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "5bac1e472735461086b47098b576e470"
format_stamp: "Formatted at 2026-08-08 12:57:01 on dist-test-slave-tk9z"
server_key: "2f8523a8852f542d938736c5db090991"
server_key_iv: "b5ad8b1cc59e7baab10a155515f76fe9"
server_key_version: "encryptionkey@0"
I20260808 12:57:01.199996 30372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:01.214486 30372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:01.218973 30372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:01.219295 30372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:01.220314 30372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:01.221801 30372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:01.221915 30372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:01.222074 30372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:01.222154 30372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:01.257324 30372 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:41951
I20260808 12:57:01.257370 30560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:41951 every 8 connection(s)
I20260808 12:57:01.258750 30372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:57:01.268735 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 30372
I20260808 12:57:01.268977 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:57:01.269935 25244 external_mini_cluster.cc:1428] Setting key 05af0982af057e07b9ad1ceff12323bb
Aug 08 12:57:01 dist-test-slave-tk9z krb5kdc[30209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193821, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:01.280131 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:01.261414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:46849 (local address 127.24.167.62:39839)
0808 12:57:01.261862 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:01.261867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:01.262958 (+ 1091us) server_negotiation.cc:407] Connection header received
0808 12:57:01.264835 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:01.264839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:01.264911 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:01.265021 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:01.266535 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:01.267252 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:01.269242 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:01.269987 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:01.270141 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:01.274165 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:01.274170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:01.274173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:01.274207 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:01.276928 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:01.277706 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:01.277711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:01.277713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:01.277834 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:01.278291 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:01.278301 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:01.278303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:01.278628 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:01.278785 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:01.279504 (+ 719us) server_negotiation.cc:299] Negotiation successful
0808 12:57:01.279760 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":225,"threads_started":1}
I20260808 12:57:01.281102 30565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:01.261997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:39839 (local address 127.24.167.1:46849)
0808 12:57:01.262795 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:01.262832 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:01.264586 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:01.265223 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:01.265230 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:01.265764 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:01.266314 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:01.266331 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:01.267390 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:01.267394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:01.269008 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:01.269024 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:01.270238 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:01.270246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:01.270602 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:01.271366 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:01.271389 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:01.273927 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:01.277101 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:01.277109 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:01.277125 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:01.277565 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:01.277951 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:01.277957 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:01.277961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:01.278166 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:01.278760 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:01.278769 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:01.279127 (+ 358us) client_negotiation.cc:769] Sending connection context
0808 12:57:01.279388 (+ 261us) client_negotiation.cc:240] Negotiation successful
0808 12:57:01.279944 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":382,"threads_started":1}
I20260808 12:57:01.282795 30561 heartbeater.cc:347] Connected to a master server at 127.24.167.62:39839
I20260808 12:57:01.283149 30561 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:01.284122 30561 heartbeater.cc:511] Master 127.24.167.62:39839 requested a full tablet report, sending...
I20260808 12:57:01.286614 30280 ts_manager.cc:194] Registered new tserver with Master: 5bac1e472735461086b47098b576e470 (127.24.167.1:41951)
I20260808 12:57:01.288861 30280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:46849
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 12:57:01.321020 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:39839
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:01.661629 30568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:01.661976 30568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:01.662159 30568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:01.669384 30568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:01.669539 30568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:01.669766 30568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:57:01.679886 30568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:39839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:01.681293 30568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:01.682766 30568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:01.696085 30573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:01.696060 30574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:01.696092 30576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:01.697374 30568 server_base.cc:1034] running on GCE node
I20260808 12:57:01.698002 30568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:01.699010 30568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:01.700237 30568 hybrid_clock.cc:648] HybridClock initialized: now 1786193821700180 us; error 52 us; skew 500 ppm
I20260808 12:57:01.700627 30568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:01 dist-test-slave-tk9z krb5kdc[30209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193821, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:01.703876 30568 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 12:57:01.705365 30568 webserver.cc:460] Webserver started at http://127.24.167.2:45775/ using document root <none> and password file <none>
I20260808 12:57:01.706044 30568 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:01.706179 30568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:01.706563 30568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:01.709481 30568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "e7b5faa833094f3d96eb7bd78cd6e0df"
format_stamp: "Formatted at 2026-08-08 12:57:01 on dist-test-slave-tk9z"
server_key: "ecaf7a6b1bd115659b405d9349e53d68"
server_key_iv: "58c03fd90afbb17a45acdf29d48d2796"
server_key_version: "encryptionkey@0"
I20260808 12:57:01.710299 30568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "e7b5faa833094f3d96eb7bd78cd6e0df"
format_stamp: "Formatted at 2026-08-08 12:57:01 on dist-test-slave-tk9z"
server_key: "ecaf7a6b1bd115659b405d9349e53d68"
server_key_iv: "58c03fd90afbb17a45acdf29d48d2796"
server_key_version: "encryptionkey@0"
I20260808 12:57:01.715538 30568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 12:57:01.718972 30585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:01.720218 30568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 12:57:01.720386 30568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "e7b5faa833094f3d96eb7bd78cd6e0df"
format_stamp: "Formatted at 2026-08-08 12:57:01 on dist-test-slave-tk9z"
server_key: "ecaf7a6b1bd115659b405d9349e53d68"
server_key_iv: "58c03fd90afbb17a45acdf29d48d2796"
server_key_version: "encryptionkey@0"
I20260808 12:57:01.720628 30568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:01.730950 30568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:01.735070 30568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:01.735370 30568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:01.736357 30568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:01.737704 30568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:01.737777 30568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:01.737950 30568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:01.738003 30568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:01.772470 30568 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:38887
I20260808 12:57:01.772514 30769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:38887 every 8 connection(s)
I20260808 12:57:01.773830 30568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:57:01.782014 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 30568
I20260808 12:57:01.782197 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:57:01.782841 25244 external_mini_cluster.cc:1428] Setting key c685504131fb3f4fb16a77b963cf1742
Aug 08 12:57:01 dist-test-slave-tk9z krb5kdc[30209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193821, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:01.795145 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:01.776427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:33173 (local address 127.24.167.62:39839)
0808 12:57:01.776616 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:01.776620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:01.777734 (+ 1114us) server_negotiation.cc:407] Connection header received
0808 12:57:01.779627 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:01.779633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:01.779724 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:01.779834 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:01.781298 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:01.782123 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:01.784146 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:01.784842 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:01.784997 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:01.789258 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:01.789263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:01.789267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:01.789299 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:01.791992 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:01.792750 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:01.792754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:01.792756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:01.792883 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:01.793345 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:01.793349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:01.793351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:01.793625 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:01.793785 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:01.794542 (+ 757us) server_negotiation.cc:299] Negotiation successful
0808 12:57:01.794784 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 12:57:01.796175 30773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:01.776900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:39839 (local address 127.24.167.2:33173)
0808 12:57:01.777587 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:01.777622 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:01.779358 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:01.780066 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:01.780075 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:01.780584 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:01.781130 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:01.781147 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:01.782305 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:01.782312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:01.784000 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:01.784013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:01.785101 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:01.785108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:01.785406 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:01.786214 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:01.786250 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:01.789021 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:01.792142 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:01.792151 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:01.792168 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:01.792608 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:01.793002 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:01.793008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:01.793012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:01.793212 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:01.793767 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:01.793775 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:01.794139 (+ 364us) client_negotiation.cc:769] Sending connection context
0808 12:57:01.794433 (+ 294us) client_negotiation.cc:240] Negotiation successful
0808 12:57:01.794998 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":270,"threads_started":1}
I20260808 12:57:01.797865 30770 heartbeater.cc:347] Connected to a master server at 127.24.167.62:39839
I20260808 12:57:01.798207 30770 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:01.799031 30770 heartbeater.cc:511] Master 127.24.167.62:39839 requested a full tablet report, sending...
I20260808 12:57:01.800587 30280 ts_manager.cc:194] Registered new tserver with Master: e7b5faa833094f3d96eb7bd78cd6e0df (127.24.167.2:38887)
I20260808 12:57:01.801443 30280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:33173
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 12:57:01.833179 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:39839
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:02.177976 30779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:02.178282 30779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:02.178436 30779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:02.185772 30779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:02.185890 30779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:02.186012 30779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:57:02.196264 30779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:39839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:02.197574 30779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:02.198940 30779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:02.212323 30785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:02.212419 30787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:02.212392 30791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:02.213639 30779 server_base.cc:1034] running on GCE node
I20260808 12:57:02.214322 30779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:02.215386 30779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:02.216636 30779 hybrid_clock.cc:648] HybridClock initialized: now 1786193822216570 us; error 58 us; skew 500 ppm
I20260808 12:57:02.216995 30779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:02 dist-test-slave-tk9z krb5kdc[30209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193822, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:02.220412 30779 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 12:57:02.222117 30779 webserver.cc:460] Webserver started at http://127.24.167.3:40933/ using document root <none> and password file <none>
I20260808 12:57:02.222949 30779 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:02.223107 30779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:02.223454 30779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:02.226558 30779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "8140fc50dcbf413fbeea32b6390019d9"
format_stamp: "Formatted at 2026-08-08 12:57:02 on dist-test-slave-tk9z"
server_key: "327668b2397c46f32c17dc8ebc2581d5"
server_key_iv: "b6f6ec906492fc4044d64f0d417b42eb"
server_key_version: "encryptionkey@0"
I20260808 12:57:02.227387 30779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "8140fc50dcbf413fbeea32b6390019d9"
format_stamp: "Formatted at 2026-08-08 12:57:02 on dist-test-slave-tk9z"
server_key: "327668b2397c46f32c17dc8ebc2581d5"
server_key_iv: "b6f6ec906492fc4044d64f0d417b42eb"
server_key_version: "encryptionkey@0"
I20260808 12:57:02.232965 30779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 12:57:02.236729 30806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:02.238184 30779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 12:57:02.238417 30779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "8140fc50dcbf413fbeea32b6390019d9"
format_stamp: "Formatted at 2026-08-08 12:57:02 on dist-test-slave-tk9z"
server_key: "327668b2397c46f32c17dc8ebc2581d5"
server_key_iv: "b6f6ec906492fc4044d64f0d417b42eb"
server_key_version: "encryptionkey@0"
I20260808 12:57:02.238679 30779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:02.254774 30779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:02.259377 30779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:02.259697 30779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:02.260756 30779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:02.262203 30779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:02.262290 30779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:02.262522 30779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:02.262614 30779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:02.293218 30561 heartbeater.cc:503] Master 127.24.167.62:39839 was elected leader, sending a full tablet report...
I20260808 12:57:02.297730 30779 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:41411
I20260808 12:57:02.297771 30974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:41411 every 8 connection(s)
I20260808 12:57:02.299239 30779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:57:02.306455 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 30779
I20260808 12:57:02.306639 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:57:02.307206 25244 external_mini_cluster.cc:1428] Setting key 185c429813566cd9063df6a4960fabff
Aug 08 12:57:02 dist-test-slave-tk9z krb5kdc[30209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193822, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:02.319631 30979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:02.301770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:52989 (local address 127.24.167.62:39839)
0808 12:57:02.302207 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:02.302212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:02.303239 (+ 1027us) server_negotiation.cc:407] Connection header received
0808 12:57:02.305333 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:02.305339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:02.305432 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:02.305549 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:02.307650 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:02.308802 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:02.310508 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:02.310915 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:02.311023 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:02.314437 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:02.314458 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:02.314462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:02.314494 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:02.316936 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:02.317622 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:02.317625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:02.317628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:02.317750 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:02.318123 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:02.318126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:02.318128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:02.318463 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:02.318577 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:02.319209 (+ 632us) server_negotiation.cc:299] Negotiation successful
0808 12:57:02.319374 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":245,"threads_started":1}
I20260808 12:57:02.320848 30980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:02.302204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:39839 (local address 127.24.167.3:52989)
0808 12:57:02.303078 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:02.303117 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:02.305076 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:02.305783 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:02.305791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:02.306631 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:02.307460 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:02.307488 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:02.308978 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:02.308985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:02.310312 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:02.310328 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:02.311124 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:02.311129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:02.311329 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:02.312019 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:02.312042 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:02.314195 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:02.317068 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:02.317074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:02.317086 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:02.317513 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:02.317837 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:02.317840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:02.317842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:02.318024 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:02.318570 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:02.318576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:02.318881 (+ 305us) client_negotiation.cc:769] Sending connection context
0808 12:57:02.319110 (+ 229us) client_negotiation.cc:240] Negotiation successful
0808 12:57:02.319493 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":374,"threads_started":1}
I20260808 12:57:02.322340 30975 heartbeater.cc:347] Connected to a master server at 127.24.167.62:39839
I20260808 12:57:02.322713 30975 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:02.323447 30975 heartbeater.cc:511] Master 127.24.167.62:39839 requested a full tablet report, sending...
I20260808 12:57:02.324858 30280 ts_manager.cc:194] Registered new tserver with Master: 8140fc50dcbf413fbeea32b6390019d9 (127.24.167.3:41411)
I20260808 12:57:02.325712 30280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:52989
I20260808 12:57:02.335177 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 12:57:02 dist-test-slave-tk9z krb5kdc[30209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 12:57:02 dist-test-slave-tk9z krb5kdc[30209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:57:02.387933 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:02.372873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54848 (local address 127.24.167.1:41951)
0808 12:57:02.373643 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:02.373650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:02.373681 (+ 31us) server_negotiation.cc:407] Connection header received
0808 12:57:02.373747 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:02.373754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:02.373993 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:02.374127 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:02.375564 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:02.376587 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:02.377772 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:02.378319 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:02.378522 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:02.382319 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:02.382330 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:02.382347 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:02.382451 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:02.385349 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:02.385944 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:02.385951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:02.385953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:02.386074 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:02.386470 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:02.386473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:02.386475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:02.386821 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:02.387060 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:02.387297 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 12:57:02.387518 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":455,"threads_started":1}
Aug 08 12:57:02 dist-test-slave-tk9z krb5kdc[30209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:02.407116 30979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:02.392242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.24.167.62:39839)
0808 12:57:02.392497 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:02.392504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:02.392675 (+ 171us) server_negotiation.cc:407] Connection header received
0808 12:57:02.392815 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:02.392819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:02.392887 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:02.392986 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:02.394285 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:02.395069 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:02.396324 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:02.396774 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:02.396896 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:02.401216 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:02.401222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:02.401225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:02.401269 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:02.404194 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:02.404951 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:02.404959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:02.404966 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:02.405104 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:02.405515 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:02.405521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:02.405529 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:02.405908 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:02.406193 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:02.406552 (+ 359us) server_negotiation.cc:299] Negotiation successful
0808 12:57:02.406796 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260808 12:57:02.407716 30280 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32980
I20260808 12:57:02.408782 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 30372
I20260808 12:57:02.421953 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:02.409859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54852 (local address 127.24.167.1:41951)
0808 12:57:02.410113 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:02.410119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:02.410347 (+ 228us) server_negotiation.cc:407] Connection header received
0808 12:57:02.410548 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:02.410552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:02.410619 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:02.410725 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:02.411968 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:02.412671 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:02.413898 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:02.414353 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:02.414570 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:02.416791 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:02.416797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:02.416801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:02.416834 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:02.419390 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:02.420041 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:02.420049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:02.420053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:02.420195 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:02.420563 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:02.420568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:02.420573 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:02.420843 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:02.421091 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:02.421395 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 12:57:02.421626 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 12:57:02.422675 30553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:02.505383 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 30372
I20260808 12:57:02.529309 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 30568
Aug 08 12:57:02 dist-test-slave-tk9z krb5kdc[30209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 12:57:02.545676 31001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:02.530328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.24.167.2:38887)
0808 12:57:02.530931 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:02.530938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:02.530963 (+ 25us) server_negotiation.cc:407] Connection header received
0808 12:57:02.531024 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:02.531031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:02.531239 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:02.531369 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:02.532634 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:02.533675 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:02.535066 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:02.535679 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:02.535802 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:02.540049 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:02.540061 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:02.540077 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:02.540113 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:02.543217 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:02.543830 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:02.543837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:02.543839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:02.543973 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:02.544288 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:02.544291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:02.544293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:02.544630 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:02.544871 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:02.545087 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 12:57:02.545277 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":301,"threads_started":1}
I20260808 12:57:02.547016 30762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:02.629873 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 30568
I20260808 12:57:02.655759 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 30779
Aug 08 12:57:02 dist-test-slave-tk9z krb5kdc[30209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 12:57:02.673808 31004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:02.656843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.24.167.3:41411)
0808 12:57:02.657530 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:02.657538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:02.657568 (+ 30us) server_negotiation.cc:407] Connection header received
0808 12:57:02.657636 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:02.657643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:02.657864 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:02.658073 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:02.659515 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:02.660589 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:02.662283 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:02.663024 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:02.663184 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:02.667751 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:02.667762 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:02.667779 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:02.667816 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:02.670797 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:02.671526 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:02.671542 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:02.671545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:02.671687 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:02.672074 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:02.672079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:02.672084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:02.672470 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:02.672836 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:02.673161 (+ 325us) server_negotiation.cc:299] Negotiation successful
0808 12:57:02.673442 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":356,"threads_started":1}
I20260808 12:57:02.675184 30969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:02.758942 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 30779
I20260808 12:57:02.788010 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 30227
I20260808 12:57:02.789287 30324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:02.893198 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 30227
2026-08-08T12:57:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4262 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:57:03 dist-test-slave-tk9z krb5kdc[31011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:57:03 dist-test-slave-tk9z krb5kdc[31011](info): set up 2 sockets
Aug 08 12:57:03 dist-test-slave-tk9z krb5kdc[31011](info): commencing operation
krb5kdc: starting...
W20260808 12:57:04.245108 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.299s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:57:04 dist-test-slave-tk9z krb5kdc[31011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:57:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:57:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 12:57:04.469890 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:46529
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:46529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:04.804358 31031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:04.804646 31031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:04.804824 31031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:04.811998 31031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:57:04.812115 31031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:04.812165 31031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:04.812253 31031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:57:04.812340 31031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:57:04.822206 31031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:46529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:46529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:04.823578 31031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:04.824960 31031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:04.837081 31038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:04.837203 31037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:04.837733 31041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:04.838136 31031 server_base.cc:1034] running on GCE node
I20260808 12:57:04.838969 31031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:04.840487 31031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:04.841728 31031 hybrid_clock.cc:648] HybridClock initialized: now 1786193824841694 us; error 45 us; skew 500 ppm
I20260808 12:57:04.842087 31031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:04 dist-test-slave-tk9z krb5kdc[31011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193824, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:04.845369 31031 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:57:04.846556 31031 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:04.846652 31031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:04.846930 31031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:04.849902 31031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "416340ed073d4d798bac31a8b1c49124"
format_stamp: "Formatted at 2026-08-08 12:57:04 on dist-test-slave-tk9z"
server_key: "9c55a4eeae5b55718acf367345756a3a"
server_key_iv: "1b36d9916ce999eeefb429cc65641786"
server_key_version: "encryptionkey@0"
I20260808 12:57:04.850862 31031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "416340ed073d4d798bac31a8b1c49124"
format_stamp: "Formatted at 2026-08-08 12:57:04 on dist-test-slave-tk9z"
server_key: "9c55a4eeae5b55718acf367345756a3a"
server_key_iv: "1b36d9916ce999eeefb429cc65641786"
server_key_version: "encryptionkey@0"
I20260808 12:57:04.856069 31031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 12:57:04.859516 31050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:04.860898 31031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 12:57:04.861042 31031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "416340ed073d4d798bac31a8b1c49124"
format_stamp: "Formatted at 2026-08-08 12:57:04 on dist-test-slave-tk9z"
server_key: "9c55a4eeae5b55718acf367345756a3a"
server_key_iv: "1b36d9916ce999eeefb429cc65641786"
server_key_version: "encryptionkey@0"
I20260808 12:57:04.861240 31031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:04.878517 31031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:04.883107 31031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:04.883409 31031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:04.901333 31031 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:46529
I20260808 12:57:04.901352 31129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:46529 every 8 connection(s)
I20260808 12:57:04.902318 31031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:57:04.906504 31130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:04.911799 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 31031
I20260808 12:57:04.912032 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:57:04.912968 25244 external_mini_cluster.cc:1428] Setting key b67f8ec484717f5ba0e51c596f5f4010
I20260808 12:57:04.924961 31130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124: Bootstrap starting.
I20260808 12:57:04.928956 31130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:04.930068 31130 log.cc:824] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:04.932997 31130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124: No bootstrap required, opened a new log
Aug 08 12:57:04 dist-test-slave-tk9z krb5kdc[31011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:04.941134 31130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416340ed073d4d798bac31a8b1c49124" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46529 } }
I20260808 12:57:04.941545 31130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:04.941664 31130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416340ed073d4d798bac31a8b1c49124, State: Initialized, Role: FOLLOWER
I20260808 12:57:04.941967 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:04.915761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.24.167.62:46529)
0808 12:57:04.920213 (+ 4452us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:04.920234 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:04.920286 (+ 52us) server_negotiation.cc:407] Connection header received
0808 12:57:04.922871 (+ 2585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:04.922902 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:04.923461 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:04.924583 (+ 1122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:04.926016 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:04.927651 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:04.929085 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:04.929777 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:04.929985 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:04.934286 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:04.934294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:04.934308 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:04.934343 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:04.937539 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:04.938253 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:04.938271 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:04.938277 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:04.938515 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:04.938909 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:04.938918 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:04.938921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:04.939506 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:04.939845 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:04.940377 (+ 532us) server_negotiation.cc:299] Negotiation successful
0808 12:57:04.940775 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4093,"thread_start_us":3828,"threads_started":1}
I20260808 12:57:04.942387 31130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416340ed073d4d798bac31a8b1c49124" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46529 } }
I20260808 12:57:04.942695 31130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:57:04.942842 31130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:57:04.943012 31130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:04.945636 31130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416340ed073d4d798bac31a8b1c49124" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46529 } }
I20260808 12:57:04.946062 31130 leader_election.cc:302] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 416340ed073d4d798bac31a8b1c49124; no voters:
I20260808 12:57:04.946702 31130 leader_election.cc:288] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:57:04.946861 31142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:04.947708 31142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [term 1 LEADER]: Becoming Leader. State: Replica: 416340ed073d4d798bac31a8b1c49124, State: Running, Role: LEADER
I20260808 12:57:04.948263 31142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416340ed073d4d798bac31a8b1c49124" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46529 } }
I20260808 12:57:04.948921 31130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:57:04.952390 31152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 416340ed073d4d798bac31a8b1c49124. Latest consensus state: current_term: 1 leader_uuid: "416340ed073d4d798bac31a8b1c49124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416340ed073d4d798bac31a8b1c49124" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46529 } } }
I20260808 12:57:04.952806 31152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:04.953697 31163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:57:04.953549 31150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "416340ed073d4d798bac31a8b1c49124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416340ed073d4d798bac31a8b1c49124" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46529 } } }
I20260808 12:57:04.953867 31150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:04.959118 31163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:57:04.968040 31163 catalog_manager.cc:1347] Generated new cluster ID: 603f5642518d4a8497024ce5b067ce98
I20260808 12:57:04.968155 31163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:57:05.003758 31163 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:57:05.005244 31163 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:57:05.016104 31163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124: Generated new TSK 0
I20260808 12:57:05.017014 31163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 12:57:05.085197 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:46529
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:05.431746 31176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:05.432114 31176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:05.432277 31176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:05.439836 31176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:05.439960 31176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:05.440128 31176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:57:05.450510 31176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:46529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:05.451937 31176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:05.453299 31176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:05.466315 31184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:05.466427 31185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:05.466503 31190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:05.467101 31176 server_base.cc:1034] running on GCE node
I20260808 12:57:05.467738 31176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:05.468645 31176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:05.469880 31176 hybrid_clock.cc:648] HybridClock initialized: now 1786193825469833 us; error 55 us; skew 500 ppm
I20260808 12:57:05.470243 31176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:05 dist-test-slave-tk9z krb5kdc[31011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193825, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:05.473438 31176 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 12:57:05.474589 31176 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:05.474682 31176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:05.474949 31176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:05.477752 31176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "10c8cfc041c342359dc27a53eb462d2f"
format_stamp: "Formatted at 2026-08-08 12:57:05 on dist-test-slave-tk9z"
server_key: "a45ad623fdf22a401189f9eed79b13c3"
server_key_iv: "c39cd7b2556d2d50192f9e05fd7eb286"
server_key_version: "encryptionkey@0"
I20260808 12:57:05.478605 31176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "10c8cfc041c342359dc27a53eb462d2f"
format_stamp: "Formatted at 2026-08-08 12:57:05 on dist-test-slave-tk9z"
server_key: "a45ad623fdf22a401189f9eed79b13c3"
server_key_iv: "c39cd7b2556d2d50192f9e05fd7eb286"
server_key_version: "encryptionkey@0"
I20260808 12:57:05.483664 31176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 12:57:05.487028 31197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:05.488379 31176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 12:57:05.488556 31176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "10c8cfc041c342359dc27a53eb462d2f"
format_stamp: "Formatted at 2026-08-08 12:57:05 on dist-test-slave-tk9z"
server_key: "a45ad623fdf22a401189f9eed79b13c3"
server_key_iv: "c39cd7b2556d2d50192f9e05fd7eb286"
server_key_version: "encryptionkey@0"
I20260808 12:57:05.488770 31176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:05.506410 31176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:05.510622 31176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:05.510955 31176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:05.511981 31176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:05.513335 31176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:05.513428 31176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:05.513574 31176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:05.513681 31176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:05.547717 31176 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:40119
I20260808 12:57:05.547771 31359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:40119 every 8 connection(s)
I20260808 12:57:05.548628 31176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:57:05.552220 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 31176
I20260808 12:57:05.552459 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:57:05.553333 25244 external_mini_cluster.cc:1428] Setting key 8e70fc09d7d8006a3ba3d3c4fdb139e9
Aug 08 12:57:05 dist-test-slave-tk9z krb5kdc[31011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193825, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:05.574263 31364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:05.551371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:60057 (local address 127.24.167.62:46529)
0808 12:57:05.551899 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:05.551906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:05.552937 (+ 1031us) server_negotiation.cc:407] Connection header received
0808 12:57:05.555384 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:05.555389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:05.555471 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:05.555585 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:05.557047 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:05.558239 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:05.559792 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:05.560523 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:05.560675 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:05.567039 (+ 6364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:05.567048 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:05.567056 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:05.567103 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:05.570820 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:05.571554 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:05.571562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:05.571569 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:05.571721 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:05.572161 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:05.572168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:05.572173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:05.572581 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:05.572762 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:05.573489 (+ 727us) server_negotiation.cc:299] Negotiation successful
0808 12:57:05.573761 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":2610,"threads_started":1}
I20260808 12:57:05.574836 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:05.551838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:46529 (local address 127.24.167.1:60057)
0808 12:57:05.552751 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:05.552802 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:05.555077 (+ 2275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:05.555794 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:05.555802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:05.556304 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:05.556868 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:05.556889 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:05.558444 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:05.558448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:05.559639 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:05.559652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:05.560745 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:05.560749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:05.560931 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:05.564415 (+ 3484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:05.564489 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:05.566786 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:05.570960 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:05.570968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:05.570980 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:05.571410 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:05.571831 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:05.571834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:05.571836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:05.572016 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:05.572723 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:05.572729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:05.573068 (+ 339us) client_negotiation.cc:769] Sending connection context
0808 12:57:05.573292 (+ 224us) client_negotiation.cc:240] Negotiation successful
0808 12:57:05.573689 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":363,"threads_started":1}
I20260808 12:57:05.576479 31360 heartbeater.cc:347] Connected to a master server at 127.24.167.62:46529
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
I20260808 12:57:05.576823 31360 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:05.577500 31360 heartbeater.cc:511] Master 127.24.167.62:46529 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
I20260808 12:57:05.579929 31076 ts_manager.cc:194] Registered new tserver with Master: 10c8cfc041c342359dc27a53eb462d2f (127.24.167.1:40119)
I20260808 12:57:05.582144 31076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:60057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 12:57:05.605580 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:46529
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:05.945469 31369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:05.945788 31369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:05.945968 31369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:05.953610 31369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:05.953778 31369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:05.953967 31369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:57:05.964133 31369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:46529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:05.965720 31369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:05.967096 31369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:05.980551 31375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:05.980576 31376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:05.981235 31381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:05.981526 31369 server_base.cc:1034] running on GCE node
I20260808 12:57:05.982152 31369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:05.983125 31369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:05.984323 31369 hybrid_clock.cc:648] HybridClock initialized: now 1786193825984276 us; error 43 us; skew 500 ppm
I20260808 12:57:05.984608 31369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:05 dist-test-slave-tk9z krb5kdc[31011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193825, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:05.987954 31369 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 12:57:05.989046 31369 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:05.989138 31369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:05.989405 31369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:05.992553 31369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "ed77721419e64e5583e4497ad6209f81"
format_stamp: "Formatted at 2026-08-08 12:57:05 on dist-test-slave-tk9z"
server_key: "d1e78b3ddc0836729520356ca9464d86"
server_key_iv: "89c446bc2d3c337a07f87f5c3829fd78"
server_key_version: "encryptionkey@0"
I20260808 12:57:05.993453 31369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "ed77721419e64e5583e4497ad6209f81"
format_stamp: "Formatted at 2026-08-08 12:57:05 on dist-test-slave-tk9z"
server_key: "d1e78b3ddc0836729520356ca9464d86"
server_key_iv: "89c446bc2d3c337a07f87f5c3829fd78"
server_key_version: "encryptionkey@0"
I20260808 12:57:05.998955 31369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 12:57:06.002485 31390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:06.003839 31369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 12:57:06.004040 31369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "ed77721419e64e5583e4497ad6209f81"
format_stamp: "Formatted at 2026-08-08 12:57:05 on dist-test-slave-tk9z"
server_key: "d1e78b3ddc0836729520356ca9464d86"
server_key_iv: "89c446bc2d3c337a07f87f5c3829fd78"
server_key_version: "encryptionkey@0"
I20260808 12:57:06.004276 31369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:06.022766 31369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:06.027164 31369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:06.027446 31369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:06.028414 31369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:06.029779 31369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:06.029877 31369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:06.030045 31369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:06.030112 31369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:06.064347 31369 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:39173
I20260808 12:57:06.064482 31563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:39173 every 8 connection(s)
I20260808 12:57:06.065356 31369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:57:06.067665 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 31369
I20260808 12:57:06.067842 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:57:06.068395 25244 external_mini_cluster.cc:1428] Setting key fbcda117f6221c58bf0a1f46836c67ac
Aug 08 12:57:06 dist-test-slave-tk9z krb5kdc[31011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193825, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:06.089169 31364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.068182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:60575 (local address 127.24.167.62:46529)
0808 12:57:06.068417 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:06.068425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:06.069897 (+ 1472us) server_negotiation.cc:407] Connection header received
0808 12:57:06.072574 (+ 2677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:06.072579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:06.072653 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:06.072754 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:06.075119 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.075845 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.078500 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.078997 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.079157 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.082997 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:06.083003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:06.083007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:06.083043 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:06.085805 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:06.086719 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:06.086723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:06.086725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:06.086866 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:06.087343 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:06.087346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:06.087348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:06.087636 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:06.087818 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:06.088554 (+ 736us) server_negotiation.cc:299] Negotiation successful
0808 12:57:06.088800 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 12:57:06.090209 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.068801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:46529 (local address 127.24.167.2:60575)
0808 12:57:06.069742 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:06.069781 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:06.072227 (+ 2446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:06.073046 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:06.073063 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:06.073916 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:06.074866 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:06.074899 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.076025 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.076033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:06.078261 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:06.078282 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.079240 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.079246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:06.079456 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.080271 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:06.080305 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:06.082714 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:06.085974 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:06.085984 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:06.086004 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:06.086569 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:06.086998 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:06.087004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:06.087008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:06.087203 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:06.087785 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:06.087794 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:06.088168 (+ 374us) client_negotiation.cc:769] Sending connection context
0808 12:57:06.088452 (+ 284us) client_negotiation.cc:240] Negotiation successful
0808 12:57:06.089029 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":403,"threads_started":1}
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
I20260808 12:57:06.091799 31565 heartbeater.cc:347] Connected to a master server at 127.24.167.62:46529
I20260808 12:57:06.092152 31565 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:06.092973 31565 heartbeater.cc:511] Master 127.24.167.62:46529 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
I20260808 12:57:06.094704 31076 ts_manager.cc:194] Registered new tserver with Master: ed77721419e64e5583e4497ad6209f81 (127.24.167.2:39173)
I20260808 12:57:06.095599 31076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:60575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 12:57:06.119557 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:46529
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:06.458701 31574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:06.458993 31574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:06.459184 31574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:06.466732 31574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:06.466863 31574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:06.467061 31574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:57:06.477020 31574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:46529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:06.478449 31574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:06.479764 31574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:06.493443 31583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:06.493566 31582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:06.493546 31590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:06.494679 31574 server_base.cc:1034] running on GCE node
I20260808 12:57:06.495334 31574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:06.496341 31574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:06.497543 31574 hybrid_clock.cc:648] HybridClock initialized: now 1786193826497500 us; error 39 us; skew 500 ppm
I20260808 12:57:06.497879 31574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:06 dist-test-slave-tk9z krb5kdc[31011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193826, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:06.501256 31574 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 12:57:06.502588 31574 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:06.502708 31574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:06.503049 31574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:06.506008 31574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "91434181b1404e75bc7e61e7a4b9cca5"
format_stamp: "Formatted at 2026-08-08 12:57:06 on dist-test-slave-tk9z"
server_key: "a90ff846d8424cbd7e290e91038fc3f2"
server_key_iv: "2b75069349a044f869f2660e5148b0a9"
server_key_version: "encryptionkey@0"
I20260808 12:57:06.506970 31574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "91434181b1404e75bc7e61e7a4b9cca5"
format_stamp: "Formatted at 2026-08-08 12:57:06 on dist-test-slave-tk9z"
server_key: "a90ff846d8424cbd7e290e91038fc3f2"
server_key_iv: "2b75069349a044f869f2660e5148b0a9"
server_key_version: "encryptionkey@0"
I20260808 12:57:06.512162 31574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 12:57:06.515583 31601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:06.516860 31574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 12:57:06.517052 31574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "91434181b1404e75bc7e61e7a4b9cca5"
format_stamp: "Formatted at 2026-08-08 12:57:06 on dist-test-slave-tk9z"
server_key: "a90ff846d8424cbd7e290e91038fc3f2"
server_key_iv: "2b75069349a044f869f2660e5148b0a9"
server_key_version: "encryptionkey@0"
I20260808 12:57:06.517297 31574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:06.543040 31574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:06.547837 31574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:06.548138 31574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:06.549207 31574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:06.550781 31574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:06.550904 31574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:06.551041 31574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:06.551127 31574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:06.585397 31360 heartbeater.cc:503] Master 127.24.167.62:46529 was elected leader, sending a full tablet report...
I20260808 12:57:06.586844 31574 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:33549
I20260808 12:57:06.586899 31768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:33549 every 8 connection(s)
I20260808 12:57:06.587947 31574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:57:06.592751 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 31574
I20260808 12:57:06.592985 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:57:06.593454 25244 external_mini_cluster.cc:1428] Setting key 8325d26cf2686697540324bb29a5e9d8
Aug 08 12:57:06 dist-test-slave-tk9z krb5kdc[31011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193826, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:06.609119 31773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.590578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:58311 (local address 127.24.167.62:46529)
0808 12:57:06.591087 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:06.591095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:06.592162 (+ 1067us) server_negotiation.cc:407] Connection header received
0808 12:57:06.594436 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:06.594442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:06.594551 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:06.594666 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:06.596453 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.597284 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.599236 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.599654 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.599755 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.603114 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:06.603119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:06.603122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:06.603154 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:06.605614 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:06.606300 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:06.606303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:06.606306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:06.606490 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:06.606906 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:06.606912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:06.606915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:06.607191 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:06.607361 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:06.608458 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0808 12:57:06.608644 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":283,"threads_started":1}
I20260808 12:57:06.610018 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.591227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:46529 (local address 127.24.167.3:58311)
0808 12:57:06.592000 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:06.592046 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:06.594082 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:06.594983 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:06.594992 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:06.595607 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:06.596251 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:06.596281 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.597433 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.597437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:06.599088 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:06.599100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.599841 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.599845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:06.600049 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.600735 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:06.600758 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:06.602914 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:06.605743 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:06.605750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:06.605760 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:06.606191 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:06.606582 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:06.606586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:06.606588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:06.606759 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:06.607343 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:06.607349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:06.607846 (+ 497us) client_negotiation.cc:769] Sending connection context
0808 12:57:06.608170 (+ 324us) client_negotiation.cc:240] Negotiation successful
0808 12:57:06.608646 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":293,"threads_started":1}
I20260808 12:57:06.611378 31769 heartbeater.cc:347] Connected to a master server at 127.24.167.62:46529
I20260808 12:57:06.611725 31769 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:06.612439 31769 heartbeater.cc:511] Master 127.24.167.62:46529 requested a full tablet report, sending...
I20260808 12:57:06.614028 31076 ts_manager.cc:194] Registered new tserver with Master: 91434181b1404e75bc7e61e7a4b9cca5 (127.24.167.3:33549)
I20260808 12:57:06.614993 31076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:58311
I20260808 12:57:06.621445 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 12:57:06.638576 31773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.626591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.24.167.62:46529)
0808 12:57:06.626759 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:06.626764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:06.627256 (+ 492us) server_negotiation.cc:407] Connection header received
0808 12:57:06.627385 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:06.627388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:06.627458 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:06.627541 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:06.628948 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.629649 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.630969 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.631374 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.631481 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.633632 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:06.633637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:06.633640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:06.633670 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:06.636273 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:06.636847 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:06.636851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:06.636853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:06.636972 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:06.637283 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:06.637286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:06.637288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:06.637598 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:06.637783 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:06.638061 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 12:57:06.638222 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 12:57:06.642778 31076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:57:06.649915 31076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:57:06.676362 31791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.663815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.24.167.3:33549)
0808 12:57:06.668219 (+ 4404us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:06.668232 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:06.668267 (+ 35us) server_negotiation.cc:407] Connection header received
0808 12:57:06.668367 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:06.668375 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:06.668643 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:06.668889 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:06.669863 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.671493 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.673621 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.675153 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.675302 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.675596 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:06.675838 (+ 242us) server_negotiation.cc:299] Negotiation successful
0808 12:57:06.676020 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4176,"thread_start_us":3991,"threads_started":1}
I20260808 12:57:06.676383 31789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.663056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:33549 (local address 127.0.0.1:49030)
0808 12:57:06.667430 (+ 4374us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:06.667450 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:06.667567 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:06.669059 (+ 1492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:06.669069 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:06.669101 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:06.669608 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:06.669624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.671630 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.671636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:06.673427 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:06.673447 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.675360 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.675366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:06.675573 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.675626 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 12:57:06.675782 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 12:57:06.675972 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4234,"thread_start_us":4076,"threads_started":1}
I20260808 12:57:06.677845 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.667830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.24.167.2:39173)
0808 12:57:06.668590 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:06.668602 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:06.669292 (+ 690us) server_negotiation.cc:407] Connection header received
0808 12:57:06.669674 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:06.669683 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:06.669895 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:06.670094 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:06.671050 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.672630 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.675244 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.676736 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.676897 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.677093 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:06.677357 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 12:57:06.677517 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":378,"threads_started":1}
I20260808 12:57:06.677969 31787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.663007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:40119 (local address 127.0.0.1:39042)
0808 12:57:06.665013 (+ 2006us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:06.665078 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:06.665289 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:06.670473 (+ 5184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:06.670478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:06.670504 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:06.670972 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:06.670984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.672998 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.673003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:06.675252 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:06.675267 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.677157 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.677163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:06.677324 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.677355 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 12:57:06.677420 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 12:57:06.677552 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1668,"mutex_wait_us":31,"thread_start_us":1499,"threads_started":1}
I20260808 12:57:06.678478 31793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.668522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:39173 (local address 127.0.0.1:40610)
0808 12:57:06.669148 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:06.669174 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:06.669403 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:06.670361 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:06.670390 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:06.670415 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:06.670875 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:06.670888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.672799 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.672805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:06.675076 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:06.675089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.676952 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.676958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:06.677149 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.677173 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 12:57:06.677300 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 12:57:06.677429 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":1,"thread_start_us":293,"threads_started":1}
I20260808 12:57:06.678726 31790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.663509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.24.167.1:40119)
0808 12:57:06.669837 (+ 6328us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:06.669844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:06.669869 (+ 25us) server_negotiation.cc:407] Connection header received
0808 12:57:06.669964 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:06.669971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:06.670192 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:06.671119 (+ 927us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:06.671387 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.672851 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.675437 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.676908 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.677698 (+ 790us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.677928 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:06.678007 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 12:57:06.678335 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5924,"thread_start_us":5760,"threads_started":1}
I20260808 12:57:06.681084 31667 tablet_service.cc:1467] Processing CreateTablet for tablet 32a31d947a5b4baabb637c7acf9274ee (DEFAULT_TABLE table=test-table [id=96b4b308fea64d8c9b47428a87d6f19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:06.682085 31462 tablet_service.cc:1467] Processing CreateTablet for tablet 32a31d947a5b4baabb637c7acf9274ee (DEFAULT_TABLE table=test-table [id=96b4b308fea64d8c9b47428a87d6f19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:06.682685 31667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a31d947a5b4baabb637c7acf9274ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:06.682232 31264 tablet_service.cc:1467] Processing CreateTablet for tablet 32a31d947a5b4baabb637c7acf9274ee (DEFAULT_TABLE table=test-table [id=96b4b308fea64d8c9b47428a87d6f19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:06.683403 31264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a31d947a5b4baabb637c7acf9274ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:06.683452 31462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a31d947a5b4baabb637c7acf9274ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:06.698103 31797 tablet_bootstrap.cc:492] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5: Bootstrap starting.
I20260808 12:57:06.700655 31794 tablet_bootstrap.cc:492] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f: Bootstrap starting.
I20260808 12:57:06.702054 31795 tablet_bootstrap.cc:492] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81: Bootstrap starting.
I20260808 12:57:06.704134 31794 tablet_bootstrap.cc:654] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:06.704262 31797 tablet_bootstrap.cc:654] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:06.705227 31794 log.cc:824] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:06.705560 31797 log.cc:824] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:06.705627 31795 tablet_bootstrap.cc:654] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:06.706840 31795 log.cc:824] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:06.708876 31797 tablet_bootstrap.cc:492] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5: No bootstrap required, opened a new log
I20260808 12:57:06.708880 31794 tablet_bootstrap.cc:492] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f: No bootstrap required, opened a new log
I20260808 12:57:06.709168 31797 ts_tablet_manager.cc:1397] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260808 12:57:06.709190 31794 ts_tablet_manager.cc:1397] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260808 12:57:06.709977 31795 tablet_bootstrap.cc:492] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81: No bootstrap required, opened a new log
I20260808 12:57:06.710253 31795 ts_tablet_manager.cc:1397] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 12:57:06.716413 31797 raft_consensus.cc:348] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91434181b1404e75bc7e61e7a4b9cca5" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 33549 } } peers { permanent_uuid: "10c8cfc041c342359dc27a53eb462d2f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 40119 } } peers { permanent_uuid: "ed77721419e64e5583e4497ad6209f81" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39173 } }
I20260808 12:57:06.716810 31797 raft_consensus.cc:374] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:06.716928 31797 raft_consensus.cc:729] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91434181b1404e75bc7e61e7a4b9cca5, State: Initialized, Role: FOLLOWER
I20260808 12:57:06.717553 31797 consensus_queue.cc:260] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91434181b1404e75bc7e61e7a4b9cca5" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 33549 } } peers { permanent_uuid: "10c8cfc041c342359dc27a53eb462d2f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 40119 } } peers { permanent_uuid: "ed77721419e64e5583e4497ad6209f81" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39173 } }
I20260808 12:57:06.718751 31769 heartbeater.cc:503] Master 127.24.167.62:46529 was elected leader, sending a full tablet report...
I20260808 12:57:06.719327 31797 ts_tablet_manager.cc:1428] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260808 12:57:06.721352 31795 raft_consensus.cc:348] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91434181b1404e75bc7e61e7a4b9cca5" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 33549 } } peers { permanent_uuid: "10c8cfc041c342359dc27a53eb462d2f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 40119 } } peers { permanent_uuid: "ed77721419e64e5583e4497ad6209f81" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39173 } }
I20260808 12:57:06.721398 31794 raft_consensus.cc:348] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91434181b1404e75bc7e61e7a4b9cca5" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 33549 } } peers { permanent_uuid: "10c8cfc041c342359dc27a53eb462d2f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 40119 } } peers { permanent_uuid: "ed77721419e64e5583e4497ad6209f81" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39173 } }
I20260808 12:57:06.721850 31795 raft_consensus.cc:374] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:06.721864 31794 raft_consensus.cc:374] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:06.721973 31795 raft_consensus.cc:729] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed77721419e64e5583e4497ad6209f81, State: Initialized, Role: FOLLOWER
I20260808 12:57:06.721992 31794 raft_consensus.cc:729] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c8cfc041c342359dc27a53eb462d2f, State: Initialized, Role: FOLLOWER
I20260808 12:57:06.722642 31795 consensus_queue.cc:260] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91434181b1404e75bc7e61e7a4b9cca5" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 33549 } } peers { permanent_uuid: "10c8cfc041c342359dc27a53eb462d2f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 40119 } } peers { permanent_uuid: "ed77721419e64e5583e4497ad6209f81" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39173 } }
I20260808 12:57:06.722664 31794 consensus_queue.cc:260] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91434181b1404e75bc7e61e7a4b9cca5" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 33549 } } peers { permanent_uuid: "10c8cfc041c342359dc27a53eb462d2f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 40119 } } peers { permanent_uuid: "ed77721419e64e5583e4497ad6209f81" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39173 } }
I20260808 12:57:06.724812 31794 ts_tablet_manager.cc:1428] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260808 12:57:06.725811 31565 heartbeater.cc:503] Master 127.24.167.62:46529 was elected leader, sending a full tablet report...
I20260808 12:57:06.726500 31795 ts_tablet_manager.cc:1428] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260808 12:57:06.749408 31802 raft_consensus.cc:482] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:57:06.749788 31802 raft_consensus.cc:504] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91434181b1404e75bc7e61e7a4b9cca5" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 33549 } } peers { permanent_uuid: "10c8cfc041c342359dc27a53eb462d2f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 40119 } } peers { permanent_uuid: "ed77721419e64e5583e4497ad6209f81" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39173 } }
I20260808 12:57:06.751792 31802 leader_election.cc:288] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10c8cfc041c342359dc27a53eb462d2f (127.24.167.1:40119), ed77721419e64e5583e4497ad6209f81 (127.24.167.2:39173)
I20260808 12:57:06.758294 31790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.752146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:33627 (local address 127.24.167.1:40119)
0808 12:57:06.752397 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:06.752406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:06.752436 (+ 30us) server_negotiation.cc:407] Connection header received
0808 12:57:06.752650 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:06.752656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:06.752735 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:06.752822 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:06.753508 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.754531 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.756329 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.757344 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.757465 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.757519 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:06.757903 (+ 384us) server_negotiation.cc:299] Negotiation successful
0808 12:57:06.758014 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 12:57:06.758334 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.752007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:40119 (local address 127.24.167.3:33627)
0808 12:57:06.752289 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:06.752315 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:06.752444 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:06.753004 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:06.753008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:06.753031 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:06.753337 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:06.753348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.754659 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.754665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:06.756169 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:06.756181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.757495 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.757501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:06.757697 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.757736 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 12:57:06.757871 (+ 135us) client_negotiation.cc:240] Negotiation successful
0808 12:57:06.758025 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"spinlock_wait_cycles":40832}
I20260808 12:57:06.759019 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.752543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:42489 (local address 127.24.167.2:39173)
0808 12:57:06.752737 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:06.752744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:06.753120 (+ 376us) server_negotiation.cc:407] Connection header received
0808 12:57:06.753288 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:06.753294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:06.753369 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:06.753473 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:06.754079 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.755034 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.756977 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.757997 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.758151 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.758226 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:06.758658 (+ 432us) server_negotiation.cc:299] Negotiation successful
0808 12:57:06.758792 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 12:57:06.759570 31805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.752074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:39173 (local address 127.24.167.3:42489)
0808 12:57:06.752967 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:06.753006 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:06.753152 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:06.753519 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:06.753523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:06.753549 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:06.753903 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:06.753913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.755188 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.755191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:06.756819 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:06.756833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.758215 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.758221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:06.758438 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.758486 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 12:57:06.759207 (+ 721us) client_negotiation.cc:240] Negotiation successful
0808 12:57:06.759305 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":1,"thread_start_us":583,"threads_started":1}
I20260808 12:57:06.759388 31291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a31d947a5b4baabb637c7acf9274ee" candidate_uuid: "91434181b1404e75bc7e61e7a4b9cca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c8cfc041c342359dc27a53eb462d2f" is_pre_election: true
I20260808 12:57:06.759955 31291 raft_consensus.cc:2454] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91434181b1404e75bc7e61e7a4b9cca5 in term 0.
I20260808 12:57:06.760557 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a31d947a5b4baabb637c7acf9274ee" candidate_uuid: "91434181b1404e75bc7e61e7a4b9cca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed77721419e64e5583e4497ad6209f81" is_pre_election: true
I20260808 12:57:06.760700 31604 leader_election.cc:302] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10c8cfc041c342359dc27a53eb462d2f, 91434181b1404e75bc7e61e7a4b9cca5; no voters:
I20260808 12:57:06.761090 31492 raft_consensus.cc:2454] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91434181b1404e75bc7e61e7a4b9cca5 in term 0.
I20260808 12:57:06.761139 31802 raft_consensus.cc:2781] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:57:06.761293 31802 raft_consensus.cc:482] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:57:06.761435 31802 raft_consensus.cc:3037] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:06.764029 31802 raft_consensus.cc:504] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91434181b1404e75bc7e61e7a4b9cca5" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 33549 } } peers { permanent_uuid: "10c8cfc041c342359dc27a53eb462d2f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 40119 } } peers { permanent_uuid: "ed77721419e64e5583e4497ad6209f81" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39173 } }
I20260808 12:57:06.764750 31802 leader_election.cc:288] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [CANDIDATE]: Term 1 election: Requested vote from peers 10c8cfc041c342359dc27a53eb462d2f (127.24.167.1:40119), ed77721419e64e5583e4497ad6209f81 (127.24.167.2:39173)
I20260808 12:57:06.765198 31291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a31d947a5b4baabb637c7acf9274ee" candidate_uuid: "91434181b1404e75bc7e61e7a4b9cca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c8cfc041c342359dc27a53eb462d2f"
I20260808 12:57:06.765230 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a31d947a5b4baabb637c7acf9274ee" candidate_uuid: "91434181b1404e75bc7e61e7a4b9cca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed77721419e64e5583e4497ad6209f81"
I20260808 12:57:06.765417 31492 raft_consensus.cc:3037] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:06.765408 31291 raft_consensus.cc:3037] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:06.767815 31291 raft_consensus.cc:2454] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91434181b1404e75bc7e61e7a4b9cca5 in term 1.
I20260808 12:57:06.767819 31492 raft_consensus.cc:2454] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91434181b1404e75bc7e61e7a4b9cca5 in term 1.
I20260808 12:57:06.768230 31604 leader_election.cc:302] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10c8cfc041c342359dc27a53eb462d2f, 91434181b1404e75bc7e61e7a4b9cca5; no voters:
I20260808 12:57:06.768537 31802 raft_consensus.cc:2781] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:06.769009 31802 raft_consensus.cc:686] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [term 1 LEADER]: Becoming Leader. State: Replica: 91434181b1404e75bc7e61e7a4b9cca5, State: Running, Role: LEADER
I20260808 12:57:06.769541 31802 consensus_queue.cc:237] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91434181b1404e75bc7e61e7a4b9cca5" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 33549 } } peers { permanent_uuid: "10c8cfc041c342359dc27a53eb462d2f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 40119 } } peers { permanent_uuid: "ed77721419e64e5583e4497ad6209f81" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39173 } }
I20260808 12:57:06.774158 31076 catalog_manager.cc:5495] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91434181b1404e75bc7e61e7a4b9cca5 (127.24.167.3). New cstate: current_term: 1 leader_uuid: "91434181b1404e75bc7e61e7a4b9cca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91434181b1404e75bc7e61e7a4b9cca5" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 33549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10c8cfc041c342359dc27a53eb462d2f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed77721419e64e5583e4497ad6209f81" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39173 } health_report { overall_health: UNKNOWN } } }
I20260808 12:57:06.799010 31791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.793550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49036 (local address 127.24.167.3:33549)
0808 12:57:06.793772 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:06.793777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:06.793798 (+ 21us) server_negotiation.cc:407] Connection header received
0808 12:57:06.793944 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:06.793950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:06.794043 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:06.794206 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:57:06.794638 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.795377 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.796723 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.797144 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.797287 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.797810 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:57:06.798349 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:57:06.798557 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:06.798639 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 12:57:06.798726 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260808 12:57:06.800954 31361 tablet.cc:2369] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:57:06.807629 31492 raft_consensus.cc:1261] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81 [term 1 FOLLOWER]: Refusing update from remote peer 91434181b1404e75bc7e61e7a4b9cca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:06.807629 31291 raft_consensus.cc:1261] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f [term 1 FOLLOWER]: Refusing update from remote peer 91434181b1404e75bc7e61e7a4b9cca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:06.808868 31802 consensus_queue.cc:1035] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c8cfc041c342359dc27a53eb462d2f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:06.809231 31807 consensus_queue.cc:1035] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed77721419e64e5583e4497ad6209f81" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 12:57:06.817487 31567 tablet.cc:2369] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:57:06.837065 31076 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:06.837322 31076 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:06.841074 31076 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 416340ed073d4d798bac31a8b1c49124: Sending DeleteTablet for 3 replicas of tablet 32a31d947a5b4baabb637c7acf9274ee
I20260808 12:57:06.842463 31462 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a31d947a5b4baabb637c7acf9274ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:06 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:40610
I20260808 12:57:06.842460 31667 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a31d947a5b4baabb637c7acf9274ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:06 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:49030
I20260808 12:57:06.842460 31264 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a31d947a5b4baabb637c7acf9274ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:06 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:39042
I20260808 12:57:06.844830 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 31176
I20260808 12:57:06.855062 31823 tablet_replica.cc:330] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81: stopping tablet replica
I20260808 12:57:06.855274 31825 tablet_replica.cc:330] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f: stopping tablet replica
I20260808 12:57:06.855988 31825 raft_consensus.cc:2229] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:06.856721 31825 raft_consensus.cc:2258] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:06.856434 31823 raft_consensus.cc:2229] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:06.857733 31822 tablet_replica.cc:330] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5: stopping tablet replica
I20260808 12:57:06.857895 31823 raft_consensus.cc:2258] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:06.858543 31822 raft_consensus.cc:2229] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:57:06.859395 31822 raft_consensus.cc:2258] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:06.859895 31825 ts_tablet_manager.cc:1905] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:06.861686 31822 ts_tablet_manager.cc:1905] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:06.861788 31823 ts_tablet_manager.cc:1905] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:06.869791 31823 ts_tablet_manager.cc:1918] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:06.870121 31823 log.cc:1196] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/32a31d947a5b4baabb637c7acf9274ee
I20260808 12:57:06.870915 31823 ts_tablet_manager.cc:1939] T 32a31d947a5b4baabb637c7acf9274ee P ed77721419e64e5583e4497ad6209f81: Deleting consensus metadata
I20260808 12:57:06.871043 31822 ts_tablet_manager.cc:1918] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:06.871379 31822 log.cc:1196] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/32a31d947a5b4baabb637c7acf9274ee
I20260808 12:57:06.872064 31822 ts_tablet_manager.cc:1939] T 32a31d947a5b4baabb637c7acf9274ee P 91434181b1404e75bc7e61e7a4b9cca5: Deleting consensus metadata
I20260808 12:57:06.873215 31057 catalog_manager.cc:4841] TS ed77721419e64e5583e4497ad6209f81 (127.24.167.2:39173): tablet 32a31d947a5b4baabb637c7acf9274ee (table test-table [id=96b4b308fea64d8c9b47428a87d6f19c]) successfully deleted
I20260808 12:57:06.874001 31057 catalog_manager.cc:4841] TS 91434181b1404e75bc7e61e7a4b9cca5 (127.24.167.3:33549): tablet 32a31d947a5b4baabb637c7acf9274ee (table test-table [id=96b4b308fea64d8c9b47428a87d6f19c]) successfully deleted
I20260808 12:57:06.877008 31825 ts_tablet_manager.cc:1918] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:06.877229 31825 log.cc:1196] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/32a31d947a5b4baabb637c7acf9274ee
Aug 08 12:57:06 dist-test-slave-tk9z krb5kdc[31011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:57:06.877853 31825 ts_tablet_manager.cc:1939] T 32a31d947a5b4baabb637c7acf9274ee P 10c8cfc041c342359dc27a53eb462d2f: Deleting consensus metadata
I20260808 12:57:06.879858 31055 catalog_manager.cc:4841] TS 10c8cfc041c342359dc27a53eb462d2f (127.24.167.1:40119): tablet 32a31d947a5b4baabb637c7acf9274ee (table test-table [id=96b4b308fea64d8c9b47428a87d6f19c]) successfully deleted
I20260808 12:57:06.886153 31790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:06.846689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39048 (local address 127.24.167.1:40119)
0808 12:57:06.846880 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:06.846887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:06.860614 (+ 13727us) server_negotiation.cc:407] Connection header received
0808 12:57:06.860960 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:06.860966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:06.861064 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:06.861151 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:06.863575 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.864659 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.866712 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:06.867388 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:06.867518 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:06.878755 (+ 11237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:06.878775 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:06.878798 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:06.878856 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:06.883037 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:06.883848 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:06.883858 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:06.883862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:06.884045 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:06.884421 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:06.884427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:06.884433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:06.884928 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:06.885267 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:06.885482 (+ 215us) server_negotiation.cc:299] Negotiation successful
0808 12:57:06.885736 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 12:57:06.887045 31353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:06.993373 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 31176
I20260808 12:57:07.020884 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 31369
Aug 08 12:57:07 dist-test-slave-tk9z krb5kdc[31011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 12:57:07.037364 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:07.021752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.24.167.2:39173)
0808 12:57:07.021936 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:07.021942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:07.022064 (+ 122us) server_negotiation.cc:407] Connection header received
0808 12:57:07.022279 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:07.022283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:07.022350 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:07.022517 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:07.023809 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:07.024590 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:07.025810 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:07.026202 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:07.026340 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:07.031085 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:07.031107 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:07.031132 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:07.031197 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:07.034514 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:07.035199 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:07.035207 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:07.035210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:07.035391 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:07.035823 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:07.035826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:07.035829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:07.036230 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:07.036510 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:07.036842 (+ 332us) server_negotiation.cc:299] Negotiation successful
0808 12:57:07.037008 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 12:57:07.038092 31559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:07.132930 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 31369
I20260808 12:57:07.164304 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 31574
Aug 08 12:57:07 dist-test-slave-tk9z krb5kdc[31011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 12:57:07.181946 31791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:07.165236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49048 (local address 127.24.167.3:33549)
0808 12:57:07.165430 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:07.165435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:07.165456 (+ 21us) server_negotiation.cc:407] Connection header received
0808 12:57:07.165691 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:07.165697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:07.165791 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:07.165905 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:07.167350 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:07.168165 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:07.169449 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:07.169946 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:07.170111 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:07.175075 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:07.175093 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:07.175111 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:07.175149 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:07.178828 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:07.179664 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:07.179672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:07.179674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:07.179836 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:07.180276 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:07.180281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:07.180285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:07.180690 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:07.181049 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:07.181337 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 12:57:07.181565 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 12:57:07.182891 31761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:07.306792 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 31574
I20260808 12:57:07.334796 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 31031
I20260808 12:57:07.335992 31124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:07.444985 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 31031
2026-08-08T12:57:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4551 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T12:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:57:07Z Disabled control of system clock
I20260808 12:57:07.527858 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:44181
--webserver_interface=127.24.167.62
--webserver_port=0
--builtin_ntp_servers=127.24.167.20:40653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:44181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 12:57:07.869632 31840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:07.869920 31840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:07.870082 31840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:07.877673 31840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:57:07.877784 31840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:07.877853 31840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:07.877940 31840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:57:07.878026 31840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:57:07.887904 31840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:44181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:44181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:07.889261 31840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:07.891196 31840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:07.903666 31856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:07.903719 31849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:07.903723 31850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:07.904743 31840 server_base.cc:1034] running on GCE node
I20260808 12:57:07.905606 31840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:07.907167 31840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:07.908398 31840 hybrid_clock.cc:648] HybridClock initialized: now 1786193827908342 us; error 52 us; skew 500 ppm
I20260808 12:57:07.908782 31840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:57:07.911167 31840 webserver.cc:460] Webserver started at http://127.24.167.62:42267/ using document root <none> and password file <none>
I20260808 12:57:07.911887 31840 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:07.912020 31840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:07.912353 31840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:07.915421 31840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "e526030406d04463bbec7904f548325b"
format_stamp: "Formatted at 2026-08-08 12:57:07 on dist-test-slave-tk9z"
server_key: "4be2f3d2c5e5cf9a2b161ca61199c2ff"
server_key_iv: "08195213a7ae697aed25e7d20fddf5a3"
server_key_version: "encryptionkey@0"
I20260808 12:57:07.916301 31840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "e526030406d04463bbec7904f548325b"
format_stamp: "Formatted at 2026-08-08 12:57:07 on dist-test-slave-tk9z"
server_key: "4be2f3d2c5e5cf9a2b161ca61199c2ff"
server_key_iv: "08195213a7ae697aed25e7d20fddf5a3"
server_key_version: "encryptionkey@0"
I20260808 12:57:07.921576 31840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 12:57:07.925300 31865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:07.926805 31840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:57:07.927013 31840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "e526030406d04463bbec7904f548325b"
format_stamp: "Formatted at 2026-08-08 12:57:07 on dist-test-slave-tk9z"
server_key: "4be2f3d2c5e5cf9a2b161ca61199c2ff"
server_key_iv: "08195213a7ae697aed25e7d20fddf5a3"
server_key_version: "encryptionkey@0"
I20260808 12:57:07.927265 31840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:07.933418 31840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:07.934267 31840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:07.934614 31840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:07.954331 31840 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:44181
I20260808 12:57:07.954386 31943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:44181 every 8 connection(s)
I20260808 12:57:07.955847 31840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:57:07.958994 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 31840
I20260808 12:57:07.959237 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:57:07.959941 25244 external_mini_cluster.cc:1428] Setting key 61c8d9f8efcfe5b0013c368c3bb3e8d5
I20260808 12:57:07.960016 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:07.969612 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:07.962143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.24.167.62:44181)
0808 12:57:07.963354 (+ 1211us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:07.963368 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:07.963425 (+ 57us) server_negotiation.cc:407] Connection header received
0808 12:57:07.965379 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:07.965409 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:07.965973 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:07.966475 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:07.967553 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:07.967562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:07.967578 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:07.967597 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:07.967682 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:07.967711 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:07.967934 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:07.968080 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 12:57:07.968352 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":429,"threads_started":1}
I20260808 12:57:07.974869 31945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b: Bootstrap starting.
I20260808 12:57:07.978250 31945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:07.979421 31945 log.cc:824] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:07.982224 31945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b: No bootstrap required, opened a new log
I20260808 12:57:07.989986 31945 raft_consensus.cc:348] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e526030406d04463bbec7904f548325b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44181 } }
I20260808 12:57:07.990461 31945 raft_consensus.cc:374] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:07.990592 31945 raft_consensus.cc:729] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e526030406d04463bbec7904f548325b, State: Initialized, Role: FOLLOWER
I20260808 12:57:07.991192 31945 consensus_queue.cc:260] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e526030406d04463bbec7904f548325b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44181 } }
I20260808 12:57:07.991446 31945 raft_consensus.cc:388] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:57:07.991598 31945 raft_consensus.cc:482] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:57:07.991794 31945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:07.994155 31945 raft_consensus.cc:504] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e526030406d04463bbec7904f548325b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44181 } }
I20260808 12:57:07.994773 31945 leader_election.cc:302] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e526030406d04463bbec7904f548325b; no voters:
I20260808 12:57:07.995389 31945 leader_election.cc:288] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:57:07.995572 31954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:07.996402 31954 raft_consensus.cc:686] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [term 1 LEADER]: Becoming Leader. State: Replica: e526030406d04463bbec7904f548325b, State: Running, Role: LEADER
I20260808 12:57:07.997000 31954 consensus_queue.cc:237] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e526030406d04463bbec7904f548325b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44181 } }
I20260808 12:57:07.997695 31945 sys_catalog.cc:564] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:57:08.000567 31955 sys_catalog.cc:455] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e526030406d04463bbec7904f548325b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e526030406d04463bbec7904f548325b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44181 } } }
I20260808 12:57:08.001001 31955 sys_catalog.cc:458] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:08.000715 31956 sys_catalog.cc:455] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e526030406d04463bbec7904f548325b. Latest consensus state: current_term: 1 leader_uuid: "e526030406d04463bbec7904f548325b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e526030406d04463bbec7904f548325b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44181 } } }
I20260808 12:57:08.001186 31956 sys_catalog.cc:458] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:08.009027 31960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:57:08.014954 31960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:57:08.025215 31960 catalog_manager.cc:1347] Generated new cluster ID: 850f2acc089a4f3db334621034481200
I20260808 12:57:08.025343 31960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:57:08.063481 31960 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:57:08.063961 31960 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:57:08.080811 31960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b: Generated new TSK 0
I20260808 12:57:08.081700 31960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 12:57:08.102036 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44181
--builtin_ntp_servers=127.24.167.20:40653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 12:57:08.459611 31976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:08.459904 31976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:08.460070 31976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:08.467327 31976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:08.467432 31976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:08.467599 31976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:57:08.476984 31976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:08.478350 31976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:08.480291 31976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:08.493887 31983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:08.493911 31984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:08.494644 31986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:08.494668 31976 server_base.cc:1034] running on GCE node
I20260808 12:57:08.495352 31976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:08.496258 31976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:08.497476 31976 hybrid_clock.cc:648] HybridClock initialized: now 1786193828497421 us; error 52 us; skew 500 ppm
I20260808 12:57:08.497810 31976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:57:08.500166 31976 webserver.cc:460] Webserver started at http://127.24.167.1:39371/ using document root <none> and password file <none>
I20260808 12:57:08.500856 31976 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:08.500989 31976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:08.501354 31976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:08.504256 31976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "a841737a78a6412fa2e04f7579b0bd04"
format_stamp: "Formatted at 2026-08-08 12:57:08 on dist-test-slave-tk9z"
server_key: "2b4340359235b9ac35733773d0eddad3"
server_key_iv: "835605e4c8635a20337fbcb6eb514e1d"
server_key_version: "encryptionkey@0"
I20260808 12:57:08.505051 31976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "a841737a78a6412fa2e04f7579b0bd04"
format_stamp: "Formatted at 2026-08-08 12:57:08 on dist-test-slave-tk9z"
server_key: "2b4340359235b9ac35733773d0eddad3"
server_key_iv: "835605e4c8635a20337fbcb6eb514e1d"
server_key_version: "encryptionkey@0"
I20260808 12:57:08.510296 31976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 12:57:08.513648 31992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:08.515112 31976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 12:57:08.515319 31976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "a841737a78a6412fa2e04f7579b0bd04"
format_stamp: "Formatted at 2026-08-08 12:57:08 on dist-test-slave-tk9z"
server_key: "2b4340359235b9ac35733773d0eddad3"
server_key_iv: "835605e4c8635a20337fbcb6eb514e1d"
server_key_version: "encryptionkey@0"
I20260808 12:57:08.515564 31976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:08.522181 31976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:08.523053 31976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:08.523361 31976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:08.524474 31976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:08.526130 31976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:08.526265 31976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:08.526495 31976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:08.526602 31976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:08.562280 31976 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:34367
I20260808 12:57:08.562330 32183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:34367 every 8 connection(s)
I20260808 12:57:08.563742 31976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:57:08.568343 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 31976
I20260808 12:57:08.568567 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:57:08.569208 25244 external_mini_cluster.cc:1428] Setting key 01696a1fb81f93861f591d59fac7f0f9
I20260808 12:57:08.572191 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44181
--builtin_ntp_servers=127.24.167.20:40653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 12:57:08.572737 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:08.566527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:43195 (local address 127.24.167.62:44181)
0808 12:57:08.566999 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:08.567004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:08.568190 (+ 1186us) server_negotiation.cc:407] Connection header received
0808 12:57:08.570316 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:08.570322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:08.570412 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:08.570527 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:08.572005 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:08.572011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:08.572016 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:08.572033 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:08.572085 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:08.572098 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:08.572227 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:08.572459 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 12:57:08.572522 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":226,"threads_started":1}
I20260808 12:57:08.574038 32190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:08.567076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:44181 (local address 127.24.167.1:43195)
0808 12:57:08.568015 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:08.568060 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:08.569956 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:08.570754 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:08.570763 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:08.570873 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:08.571697 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 12:57:08.571700 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:08.571737 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 12:57:08.571820 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:08.572197 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:08.572210 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:08.572217 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 12:57:08.573022 (+ 805us) client_negotiation.cc:240] Negotiation successful
0808 12:57:08.573214 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":281,"threads_started":1}
I20260808 12:57:08.575598 32185 heartbeater.cc:347] Connected to a master server at 127.24.167.62:44181
I20260808 12:57:08.575913 32185 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:08.576510 32185 heartbeater.cc:511] Master 127.24.167.62:44181 requested a full tablet report, sending...
I20260808 12:57:08.578948 31888 ts_manager.cc:194] Registered new tserver with Master: a841737a78a6412fa2e04f7579b0bd04 (127.24.167.1:34367)
W20260808 12:57:08.922518 32191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:08.922811 32191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:08.922978 32191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:08.930457 32191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:08.930555 32191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:08.930732 32191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:57:08.940502 32191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:08.941855 32191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:08.943677 32191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:08.957132 32199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:08.957154 32200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:08.957774 32191 server_base.cc:1034] running on GCE node
W20260808 12:57:08.958601 32203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:08.959268 32191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:08.960291 32191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:08.961511 32191 hybrid_clock.cc:648] HybridClock initialized: now 1786193828961467 us; error 40 us; skew 500 ppm
I20260808 12:57:08.961836 32191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:57:08.964202 32191 webserver.cc:460] Webserver started at http://127.24.167.2:42079/ using document root <none> and password file <none>
I20260808 12:57:08.964870 32191 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:08.964958 32191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:08.965364 32191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:08.968240 32191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "30341e372a4d47778c6b16514d056bbd"
format_stamp: "Formatted at 2026-08-08 12:57:08 on dist-test-slave-tk9z"
server_key: "532f51fe15744f59e0cde00d8e65c65b"
server_key_iv: "abf8c78ddbfc0a4b59b0cf28b484cca0"
server_key_version: "encryptionkey@0"
I20260808 12:57:08.969028 32191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "30341e372a4d47778c6b16514d056bbd"
format_stamp: "Formatted at 2026-08-08 12:57:08 on dist-test-slave-tk9z"
server_key: "532f51fe15744f59e0cde00d8e65c65b"
server_key_iv: "abf8c78ddbfc0a4b59b0cf28b484cca0"
server_key_version: "encryptionkey@0"
I20260808 12:57:08.974103 32191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 12:57:08.977555 32209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:08.979027 32191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:57:08.979209 32191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "30341e372a4d47778c6b16514d056bbd"
format_stamp: "Formatted at 2026-08-08 12:57:08 on dist-test-slave-tk9z"
server_key: "532f51fe15744f59e0cde00d8e65c65b"
server_key_iv: "abf8c78ddbfc0a4b59b0cf28b484cca0"
server_key_version: "encryptionkey@0"
I20260808 12:57:08.979415 32191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:08.985976 32191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:08.986920 32191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:08.987203 32191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:08.988128 32191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:08.989534 32191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:08.989661 32191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:08.989785 32191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:08.989876 32191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:09.048498 32191 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:33029
I20260808 12:57:09.048674 32380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:33029 every 8 connection(s)
I20260808 12:57:09.052227 32191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:57:09.058107 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 32191
I20260808 12:57:09.058481 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:57:09.060194 25244 external_mini_cluster.cc:1428] Setting key 79057bd43f5e6573cae7ca27a44fec71
I20260808 12:57:09.069550 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44181
--builtin_ntp_servers=127.24.167.20:40653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 12:57:09.077148 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:09.058625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:59775 (local address 127.24.167.62:44181)
0808 12:57:09.058878 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:09.058891 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:09.064553 (+ 5662us) server_negotiation.cc:407] Connection header received
0808 12:57:09.070755 (+ 6202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:09.070802 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:09.071011 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:09.071422 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:09.074872 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:09.074878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:09.074896 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:09.074938 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:09.075019 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:09.075049 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:09.075447 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:09.076034 (+ 587us) server_negotiation.cc:299] Negotiation successful
0808 12:57:09.076302 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 12:57:09.079413 32384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:09.060878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:44181 (local address 127.24.167.2:59775)
0808 12:57:09.064115 (+ 3237us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:09.064267 (+ 152us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:09.069774 (+ 5507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:09.071991 (+ 2217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:09.072016 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:09.072285 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:09.074015 (+ 1730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 12:57:09.074026 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:09.074172 (+ 146us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 12:57:09.074455 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:09.075357 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:09.075369 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:09.075377 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 12:57:09.076341 (+ 964us) client_negotiation.cc:240] Negotiation successful
0808 12:57:09.077165 (+ 824us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1804,"thread_start_us":1026,"threads_started":1}
I20260808 12:57:09.082690 32381 heartbeater.cc:347] Connected to a master server at 127.24.167.62:44181
I20260808 12:57:09.083731 32381 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:09.086254 32381 heartbeater.cc:511] Master 127.24.167.62:44181 requested a full tablet report, sending...
I20260808 12:57:09.090555 31888 ts_manager.cc:194] Registered new tserver with Master: 30341e372a4d47778c6b16514d056bbd (127.24.167.2:33029)
I20260808 12:57:09.582661 32185 heartbeater.cc:503] Master 127.24.167.62:44181 was elected leader, sending a full tablet report...
W20260808 12:57:10.003196 32385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:10.003895 32385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:10.004153 32385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:10.022030 32385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:10.022236 32385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:10.022589 32385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:57:10.044765 32385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:10.047627 32385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:10.051527 32385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:10.083739 32399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:10.084085 32404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:10.084020 32396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:10.085741 32385 server_base.cc:1034] running on GCE node
I20260808 12:57:10.087280 32385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:10.089363 32385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:10.092839 32385 hybrid_clock.cc:648] HybridClock initialized: now 1786193830092626 us; error 164 us; skew 500 ppm
I20260808 12:57:10.093549 32385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:57:10.096657 32381 heartbeater.cc:503] Master 127.24.167.62:44181 was elected leader, sending a full tablet report...
I20260808 12:57:10.100473 32385 webserver.cc:460] Webserver started at http://127.24.167.3:42149/ using document root <none> and password file <none>
I20260808 12:57:10.102725 32385 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:10.102953 32385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:10.103636 32385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:10.111624 32385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "b51e6255552b4d7180043920d7311c5d"
format_stamp: "Formatted at 2026-08-08 12:57:10 on dist-test-slave-tk9z"
server_key: "1a3205f66df022454f654a35a8842bab"
server_key_iv: "f05aa200df41b04608fe53dc03274733"
server_key_version: "encryptionkey@0"
I20260808 12:57:10.113862 32385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "b51e6255552b4d7180043920d7311c5d"
format_stamp: "Formatted at 2026-08-08 12:57:10 on dist-test-slave-tk9z"
server_key: "1a3205f66df022454f654a35a8842bab"
server_key_iv: "f05aa200df41b04608fe53dc03274733"
server_key_version: "encryptionkey@0"
I20260808 12:57:10.126868 32385 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.010s sys 0.001s
I20260808 12:57:10.135334 32414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:10.138821 32385 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260808 12:57:10.139179 32385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "b51e6255552b4d7180043920d7311c5d"
format_stamp: "Formatted at 2026-08-08 12:57:10 on dist-test-slave-tk9z"
server_key: "1a3205f66df022454f654a35a8842bab"
server_key_iv: "f05aa200df41b04608fe53dc03274733"
server_key_version: "encryptionkey@0"
I20260808 12:57:10.139700 32385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:10.151264 32385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:10.152570 32385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:10.153026 32385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:10.154579 32385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:10.156934 32385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:10.157094 32385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:10.157323 32385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:10.157444 32385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:10.217702 32385 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:43277
I20260808 12:57:10.217782 32577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:43277 every 8 connection(s)
I20260808 12:57:10.219846 32385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:57:10.224779 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 32385
I20260808 12:57:10.225143 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:57:10.225893 25244 external_mini_cluster.cc:1428] Setting key 30182fdc47da086f654f601f82ae0181
I20260808 12:57:10.236795 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.224269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:60629 (local address 127.24.167.62:44181)
0808 12:57:10.225326 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:10.225338 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:10.227693 (+ 2355us) server_negotiation.cc:407] Connection header received
0808 12:57:10.231582 (+ 3889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:10.231587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:10.231667 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:10.231816 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:10.234874 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:10.234903 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:10.234926 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:10.234959 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:10.235002 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:10.235089 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:10.235451 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:10.235962 (+ 511us) server_negotiation.cc:299] Negotiation successful
0808 12:57:10.236104 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":563,"threads_started":1}
I20260808 12:57:10.239071 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.225571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:44181 (local address 127.24.167.3:60629)
0808 12:57:10.227436 (+ 1865us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:10.227511 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:10.231156 (+ 3645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:10.232317 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:10.232328 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:10.232552 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:10.234106 (+ 1554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 12:57:10.234117 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:10.234244 (+ 127us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 12:57:10.234584 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:10.235368 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:10.235407 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:10.235431 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 12:57:10.236181 (+ 750us) client_negotiation.cc:240] Negotiation successful
0808 12:57:10.236662 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":534,"threads_started":1}
I20260808 12:57:10.243189 32578 heartbeater.cc:347] Connected to a master server at 127.24.167.62:44181
I20260808 12:57:10.243852 32578 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:10.245172 32578 heartbeater.cc:511] Master 127.24.167.62:44181 requested a full tablet report, sending...
I20260808 12:57:10.248230 31888 ts_manager.cc:194] Registered new tserver with Master: b51e6255552b4d7180043920d7311c5d (127.24.167.3:43277)
I20260808 12:57:10.261793 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 12:57:10.281433 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.273992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.24.167.62:44181)
0808 12:57:10.274546 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:10.274557 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:10.275847 (+ 1290us) server_negotiation.cc:407] Connection header received
0808 12:57:10.276530 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:10.276534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:10.276775 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:10.277099 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:10.279568 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:10.279586 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:10.279603 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:10.279641 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:10.279699 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:10.279741 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:10.280035 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:10.280284 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 12:57:10.280435 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260808 12:57:10.292485 31888 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:57:10.311250 31888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:57:10.365537 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.345486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.24.167.1:34367)
0808 12:57:10.349120 (+ 3634us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:10.349166 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:10.351657 (+ 2491us) server_negotiation.cc:407] Connection header received
0808 12:57:10.351918 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:10.351949 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:10.352611 (+ 662us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:10.355186 (+ 2575us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:10.360166 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:10.360188 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:10.360207 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:10.360273 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:10.360488 (+ 215us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:10.360566 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:10.363841 (+ 3275us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:10.364021 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 12:57:10.364314 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3030,"thread_start_us":2347,"threads_started":1}
I20260808 12:57:10.365790 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.344022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41288 (local address 127.24.167.3:43277)
0808 12:57:10.347416 (+ 3394us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:10.347472 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:10.349685 (+ 2213us) server_negotiation.cc:407] Connection header received
0808 12:57:10.350826 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:10.350850 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:10.351606 (+ 756us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:10.352194 (+ 588us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:10.361822 (+ 9628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:10.361872 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:10.361923 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:10.361988 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:10.362204 (+ 216us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:10.362254 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:10.363841 (+ 1587us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:10.364026 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 12:57:10.364395 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2795,"thread_start_us":2218,"threads_started":1}
I20260808 12:57:10.367440 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.345305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:34367 (local address 127.0.0.1:35636)
0808 12:57:10.350889 (+ 5584us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:10.350964 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:10.351260 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:10.353311 (+ 2051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:10.353331 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:10.353615 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:10.359131 (+ 5516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 12:57:10.359154 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:10.359265 (+ 111us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 12:57:10.359499 (+ 234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:10.361119 (+ 1620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:10.361135 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:10.361144 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 12:57:10.361763 (+ 619us) client_negotiation.cc:240] Negotiation successful
0808 12:57:10.362100 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5198,"thread_start_us":3215,"threads_started":1}
I20260808 12:57:10.368181 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.343530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:43277 (local address 127.0.0.1:41288)
0808 12:57:10.349254 (+ 5724us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:10.349459 (+ 205us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:10.349920 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:10.357553 (+ 7633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:10.357558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:10.357708 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:10.360024 (+ 2316us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 12:57:10.360045 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:10.360140 (+ 95us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 12:57:10.360209 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:10.362711 (+ 2502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:10.362735 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:10.362767 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 12:57:10.363137 (+ 370us) client_negotiation.cc:240] Negotiation successful
0808 12:57:10.363285 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4936,"thread_start_us":717,"threads_started":1}
I20260808 12:57:10.379769 32079 tablet_service.cc:1467] Processing CreateTablet for tablet fa111ce453714e21bdf2d0ab0969df36 (DEFAULT_TABLE table=test-table [id=68ae7d1113c54eacb242979c1cf5e897]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:10.381758 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.350318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:33029 (local address 127.0.0.1:39728)
0808 12:57:10.369183 (+ 18865us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:10.369238 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:10.369546 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:10.375207 (+ 5661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:10.375214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:10.375392 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:10.377784 (+ 2392us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 12:57:10.377789 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:10.377826 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 12:57:10.377979 (+ 153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:10.380050 (+ 2071us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:10.380058 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:10.380064 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 12:57:10.380614 (+ 550us) client_negotiation.cc:240] Negotiation successful
0808 12:57:10.380809 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18315,"thread_start_us":20134,"threads_started":1}
I20260808 12:57:10.382222 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.351320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.24.167.2:33029)
0808 12:57:10.363805 (+ 12485us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:10.363830 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:10.369781 (+ 5951us) server_negotiation.cc:407] Connection header received
0808 12:57:10.370022 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:10.370058 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:10.370687 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:10.371989 (+ 1302us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:10.378687 (+ 6698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:10.378723 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:10.378776 (+ 53us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:10.378898 (+ 122us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:10.379031 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:10.379088 (+ 57us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:10.379738 (+ 650us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:10.380686 (+ 948us) server_negotiation.cc:299] Negotiation successful
0808 12:57:10.381007 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12050,"thread_start_us":12470,"threads_started":1}
I20260808 12:57:10.385315 32079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa111ce453714e21bdf2d0ab0969df36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:10.384778 32488 tablet_service.cc:1467] Processing CreateTablet for tablet fa111ce453714e21bdf2d0ab0969df36 (DEFAULT_TABLE table=test-table [id=68ae7d1113c54eacb242979c1cf5e897]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:10.390053 32488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa111ce453714e21bdf2d0ab0969df36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:10.397557 32285 tablet_service.cc:1467] Processing CreateTablet for tablet fa111ce453714e21bdf2d0ab0969df36 (DEFAULT_TABLE table=test-table [id=68ae7d1113c54eacb242979c1cf5e897]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:10.402660 32285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa111ce453714e21bdf2d0ab0969df36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:10.426072 32612 tablet_bootstrap.cc:492] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04: Bootstrap starting.
I20260808 12:57:10.437659 32612 tablet_bootstrap.cc:654] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:10.440492 32612 log.cc:824] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:10.444931 32613 tablet_bootstrap.cc:492] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d: Bootstrap starting.
I20260808 12:57:10.455696 32615 tablet_bootstrap.cc:492] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd: Bootstrap starting.
I20260808 12:57:10.456398 32613 tablet_bootstrap.cc:654] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:10.456816 32612 tablet_bootstrap.cc:492] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04: No bootstrap required, opened a new log
I20260808 12:57:10.457484 32612 ts_tablet_manager.cc:1397] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.009s
I20260808 12:57:10.460975 32613 log.cc:824] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:10.469643 32615 tablet_bootstrap.cc:654] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:10.471765 32613 tablet_bootstrap.cc:492] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d: No bootstrap required, opened a new log
I20260808 12:57:10.472869 32613 ts_tablet_manager.cc:1397] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.013s
I20260808 12:57:10.473848 32615 log.cc:824] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:10.484493 32612 raft_consensus.cc:348] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e6255552b4d7180043920d7311c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43277 } } peers { permanent_uuid: "30341e372a4d47778c6b16514d056bbd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33029 } } peers { permanent_uuid: "a841737a78a6412fa2e04f7579b0bd04" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34367 } }
I20260808 12:57:10.485755 32612 raft_consensus.cc:374] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:10.486166 32612 raft_consensus.cc:729] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a841737a78a6412fa2e04f7579b0bd04, State: Initialized, Role: FOLLOWER
I20260808 12:57:10.486807 32615 tablet_bootstrap.cc:492] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd: No bootstrap required, opened a new log
I20260808 12:57:10.487488 32615 ts_tablet_manager.cc:1397] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.006s
I20260808 12:57:10.487466 32612 consensus_queue.cc:260] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e6255552b4d7180043920d7311c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43277 } } peers { permanent_uuid: "30341e372a4d47778c6b16514d056bbd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33029 } } peers { permanent_uuid: "a841737a78a6412fa2e04f7579b0bd04" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34367 } }
I20260808 12:57:10.498639 32612 ts_tablet_manager.cc:1428] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04: Time spent starting tablet: real 0.041s user 0.031s sys 0.013s
I20260808 12:57:10.511260 32613 raft_consensus.cc:348] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e6255552b4d7180043920d7311c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43277 } } peers { permanent_uuid: "30341e372a4d47778c6b16514d056bbd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33029 } } peers { permanent_uuid: "a841737a78a6412fa2e04f7579b0bd04" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34367 } }
I20260808 12:57:10.512459 32613 raft_consensus.cc:374] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:10.512670 32613 raft_consensus.cc:729] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51e6255552b4d7180043920d7311c5d, State: Initialized, Role: FOLLOWER
I20260808 12:57:10.514668 32613 consensus_queue.cc:260] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e6255552b4d7180043920d7311c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43277 } } peers { permanent_uuid: "30341e372a4d47778c6b16514d056bbd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33029 } } peers { permanent_uuid: "a841737a78a6412fa2e04f7579b0bd04" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34367 } }
I20260808 12:57:10.516220 32615 raft_consensus.cc:348] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e6255552b4d7180043920d7311c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43277 } } peers { permanent_uuid: "30341e372a4d47778c6b16514d056bbd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33029 } } peers { permanent_uuid: "a841737a78a6412fa2e04f7579b0bd04" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34367 } }
I20260808 12:57:10.517670 32615 raft_consensus.cc:374] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:10.518018 32615 raft_consensus.cc:729] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30341e372a4d47778c6b16514d056bbd, State: Initialized, Role: FOLLOWER
I20260808 12:57:10.519804 32615 consensus_queue.cc:260] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e6255552b4d7180043920d7311c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43277 } } peers { permanent_uuid: "30341e372a4d47778c6b16514d056bbd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33029 } } peers { permanent_uuid: "a841737a78a6412fa2e04f7579b0bd04" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34367 } }
I20260808 12:57:10.520864 32578 heartbeater.cc:503] Master 127.24.167.62:44181 was elected leader, sending a full tablet report...
I20260808 12:57:10.524125 32613 ts_tablet_manager.cc:1428] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d: Time spent starting tablet: real 0.051s user 0.045s sys 0.005s
I20260808 12:57:10.526259 32615 ts_tablet_manager.cc:1428] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd: Time spent starting tablet: real 0.039s user 0.033s sys 0.001s
W20260808 12:57:10.558296 32382 tablet.cc:2369] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 12:57:10.568157 32188 tablet.cc:2369] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:57:10.672477 32619 raft_consensus.cc:482] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:57:10.673058 32619 raft_consensus.cc:504] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e6255552b4d7180043920d7311c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43277 } } peers { permanent_uuid: "30341e372a4d47778c6b16514d056bbd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33029 } } peers { permanent_uuid: "a841737a78a6412fa2e04f7579b0bd04" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34367 } }
I20260808 12:57:10.675873 32619 leader_election.cc:288] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30341e372a4d47778c6b16514d056bbd (127.24.167.2:33029), a841737a78a6412fa2e04f7579b0bd04 (127.24.167.1:34367)
I20260808 12:57:10.685148 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.677260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:34653 (local address 127.24.167.1:34367)
0808 12:57:10.677676 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:10.677683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:10.677705 (+ 22us) server_negotiation.cc:407] Connection header received
0808 12:57:10.677833 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:10.677849 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:10.677978 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:10.678210 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:10.684106 (+ 5896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:10.684127 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:10.684137 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:10.684186 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:10.684245 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:10.684258 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:10.684392 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:10.684643 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 12:57:10.684752 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260808 12:57:10.685175 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.676509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:44925 (local address 127.24.167.2:33029)
0808 12:57:10.676848 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:10.676872 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:10.677860 (+ 988us) server_negotiation.cc:407] Connection header received
0808 12:57:10.678144 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:10.678166 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:10.678361 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:10.678706 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:10.684225 (+ 5519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:10.684232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:10.684238 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:10.684255 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:10.684297 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:10.684386 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:10.684603 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:10.684679 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 12:57:10.684751 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260808 12:57:10.685220 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.676380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:34367 (local address 127.24.167.3:34653)
0808 12:57:10.676756 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:10.676778 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:10.677002 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:10.678325 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:10.678334 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:10.678413 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:10.683750 (+ 5337us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 12:57:10.683752 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:10.683778 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 12:57:10.683867 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:10.684424 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:10.684436 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:10.684451 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 12:57:10.684607 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 12:57:10.684697 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260808 12:57:10.685827 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.676376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:33029 (local address 127.24.167.3:44925)
0808 12:57:10.677681 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:10.677698 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:10.677885 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:10.678707 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:10.678710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:10.678729 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:10.683876 (+ 5147us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 12:57:10.683881 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:10.683904 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 12:57:10.683995 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:10.684536 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:10.684542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:10.684555 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 12:57:10.684716 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 12:57:10.684777 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":130,"thread_start_us":529,"threads_started":1}
I20260808 12:57:10.687009 32110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa111ce453714e21bdf2d0ab0969df36" candidate_uuid: "b51e6255552b4d7180043920d7311c5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a841737a78a6412fa2e04f7579b0bd04" is_pre_election: true
I20260808 12:57:10.687358 32315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa111ce453714e21bdf2d0ab0969df36" candidate_uuid: "b51e6255552b4d7180043920d7311c5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30341e372a4d47778c6b16514d056bbd" is_pre_election: true
I20260808 12:57:10.687906 32110 raft_consensus.cc:2454] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b51e6255552b4d7180043920d7311c5d in term 0.
I20260808 12:57:10.688189 32315 raft_consensus.cc:2454] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b51e6255552b4d7180043920d7311c5d in term 0.
I20260808 12:57:10.689092 32419 leader_election.cc:302] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a841737a78a6412fa2e04f7579b0bd04, b51e6255552b4d7180043920d7311c5d; no voters:
I20260808 12:57:10.689914 32619 raft_consensus.cc:2781] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:57:10.690143 32619 raft_consensus.cc:482] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:57:10.690294 32619 raft_consensus.cc:3037] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:10.698159 32619 raft_consensus.cc:504] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e6255552b4d7180043920d7311c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43277 } } peers { permanent_uuid: "30341e372a4d47778c6b16514d056bbd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33029 } } peers { permanent_uuid: "a841737a78a6412fa2e04f7579b0bd04" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34367 } }
I20260808 12:57:10.699401 32619 leader_election.cc:288] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [CANDIDATE]: Term 1 election: Requested vote from peers 30341e372a4d47778c6b16514d056bbd (127.24.167.2:33029), a841737a78a6412fa2e04f7579b0bd04 (127.24.167.1:34367)
I20260808 12:57:10.701076 32110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa111ce453714e21bdf2d0ab0969df36" candidate_uuid: "b51e6255552b4d7180043920d7311c5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a841737a78a6412fa2e04f7579b0bd04"
I20260808 12:57:10.701018 32315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa111ce453714e21bdf2d0ab0969df36" candidate_uuid: "b51e6255552b4d7180043920d7311c5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30341e372a4d47778c6b16514d056bbd"
I20260808 12:57:10.701640 32110 raft_consensus.cc:3037] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:10.701627 32315 raft_consensus.cc:3037] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:10.712833 32110 raft_consensus.cc:2454] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b51e6255552b4d7180043920d7311c5d in term 1.
I20260808 12:57:10.712834 32315 raft_consensus.cc:2454] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b51e6255552b4d7180043920d7311c5d in term 1.
I20260808 12:57:10.714174 32419 leader_election.cc:302] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30341e372a4d47778c6b16514d056bbd, a841737a78a6412fa2e04f7579b0bd04, b51e6255552b4d7180043920d7311c5d; no voters:
I20260808 12:57:10.715368 32619 raft_consensus.cc:2781] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:10.716521 32619 raft_consensus.cc:686] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [term 1 LEADER]: Becoming Leader. State: Replica: b51e6255552b4d7180043920d7311c5d, State: Running, Role: LEADER
I20260808 12:57:10.717777 32619 consensus_queue.cc:237] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e6255552b4d7180043920d7311c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43277 } } peers { permanent_uuid: "30341e372a4d47778c6b16514d056bbd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33029 } } peers { permanent_uuid: "a841737a78a6412fa2e04f7579b0bd04" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34367 } }
W20260808 12:57:10.723176 32579 tablet.cc:2369] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:57:10.731705 31887 catalog_manager.cc:5495] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d reported cstate change: term changed from 0 to 1, leader changed from <none> to b51e6255552b4d7180043920d7311c5d (127.24.167.3). New cstate: current_term: 1 leader_uuid: "b51e6255552b4d7180043920d7311c5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51e6255552b4d7180043920d7311c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30341e372a4d47778c6b16514d056bbd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a841737a78a6412fa2e04f7579b0bd04" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34367 } health_report { overall_health: UNKNOWN } } }
I20260808 12:57:10.777431 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.775314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.24.167.3:43277)
0808 12:57:10.775491 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:10.775495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:10.775516 (+ 21us) server_negotiation.cc:407] Connection header received
0808 12:57:10.775730 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:10.775734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:10.775851 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:10.775941 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:10.776881 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:10.776885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:10.776888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:10.776904 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:10.776939 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:10.776951 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:10.777049 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:10.777181 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 12:57:10.777229 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 12:57:10.786698 32315 raft_consensus.cc:1261] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd [term 1 FOLLOWER]: Refusing update from remote peer b51e6255552b4d7180043920d7311c5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:10.786697 32110 raft_consensus.cc:1261] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04 [term 1 FOLLOWER]: Refusing update from remote peer b51e6255552b4d7180043920d7311c5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:10.787890 32623 consensus_queue.cc:1035] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a841737a78a6412fa2e04f7579b0bd04" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:10.788297 32619 consensus_queue.cc:1035] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "30341e372a4d47778c6b16514d056bbd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:10.799752 32629 mvcc.cc:205] Tried to move back new op lower bound from 7316249930889908224 to 7316249930631815168. Current Snapshot: MvccSnapshot[applied={T|T < 7316249930889908224}]
I20260808 12:57:10.801556 32633 mvcc.cc:205] Tried to move back new op lower bound from 7316249930889908224 to 7316249930631815168. Current Snapshot: MvccSnapshot[applied={T|T < 7316249930889908224}]
I20260808 12:57:10.814709 31888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:10.815021 31888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:10.818765 31888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e526030406d04463bbec7904f548325b: Sending DeleteTablet for 3 replicas of tablet fa111ce453714e21bdf2d0ab0969df36
I20260808 12:57:10.820184 32488 tablet_service.cc:1514] Processing DeleteTablet for tablet fa111ce453714e21bdf2d0ab0969df36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:10 UTC) from {username='slave'} at 127.0.0.1:41288
I20260808 12:57:10.820158 32285 tablet_service.cc:1514] Processing DeleteTablet for tablet fa111ce453714e21bdf2d0ab0969df36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:10 UTC) from {username='slave'} at 127.0.0.1:39728
I20260808 12:57:10.820242 32079 tablet_service.cc:1514] Processing DeleteTablet for tablet fa111ce453714e21bdf2d0ab0969df36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:10 UTC) from {username='slave'} at 127.0.0.1:35636
I20260808 12:57:10.825188 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 31976
I20260808 12:57:10.832007 32640 tablet_replica.cc:330] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d: stopping tablet replica
I20260808 12:57:10.832670 32640 raft_consensus.cc:2229] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:57:10.833649 32640 raft_consensus.cc:2258] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:10.834334 32642 tablet_replica.cc:330] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd: stopping tablet replica
I20260808 12:57:10.834622 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.826722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.24.167.1:34367)
0808 12:57:10.826922 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:10.826929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:10.830924 (+ 3995us) server_negotiation.cc:407] Connection header received
0808 12:57:10.831104 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:10.831109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:10.831222 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:10.831325 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:10.833502 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:10.833512 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:10.833520 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:10.833547 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:10.833601 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:10.833622 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:10.833760 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:10.834049 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 12:57:10.834135 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 12:57:10.834847 32642 raft_consensus.cc:2229] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:10.835399 32642 raft_consensus.cc:2258] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:10.835706 32176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:10.838359 32640 ts_tablet_manager.cc:1905] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:10.840040 32642 ts_tablet_manager.cc:1905] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:10.847509 32640 ts_tablet_manager.cc:1918] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:10.847751 32640 log.cc:1196] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/fa111ce453714e21bdf2d0ab0969df36
I20260808 12:57:10.848466 32640 ts_tablet_manager.cc:1939] T fa111ce453714e21bdf2d0ab0969df36 P b51e6255552b4d7180043920d7311c5d: Deleting consensus metadata
I20260808 12:57:10.851110 31867 catalog_manager.cc:4841] TS b51e6255552b4d7180043920d7311c5d (127.24.167.3:43277): tablet fa111ce453714e21bdf2d0ab0969df36 (table test-table [id=68ae7d1113c54eacb242979c1cf5e897]) successfully deleted
I20260808 12:57:10.851750 32642 ts_tablet_manager.cc:1918] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:10.851961 32642 log.cc:1196] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/fa111ce453714e21bdf2d0ab0969df36
I20260808 12:57:10.852514 32642 ts_tablet_manager.cc:1939] T fa111ce453714e21bdf2d0ab0969df36 P 30341e372a4d47778c6b16514d056bbd: Deleting consensus metadata
I20260808 12:57:10.854450 31867 catalog_manager.cc:4841] TS 30341e372a4d47778c6b16514d056bbd (127.24.167.2:33029): tablet fa111ce453714e21bdf2d0ab0969df36 (table test-table [id=68ae7d1113c54eacb242979c1cf5e897]) successfully deleted
I20260808 12:57:10.939838 32643 tablet_replica.cc:330] T fa111ce453714e21bdf2d0ab0969df36 P a841737a78a6412fa2e04f7579b0bd04: stopping tablet replica
I20260808 12:57:10.939888 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 31976
W20260808 12:57:10.967586 31870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.167.1:34367 (error 108)
I20260808 12:57:10.968420 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 32191
I20260808 12:57:10.972265 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.969280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39732 (local address 127.24.167.2:33029)
0808 12:57:10.969456 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:10.969462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:10.969487 (+ 25us) server_negotiation.cc:407] Connection header received
0808 12:57:10.969715 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:10.969721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:10.969849 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:10.969959 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:10.971234 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:10.971239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:10.971245 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:10.971264 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:10.971307 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:10.971322 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:10.971436 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:10.971915 (+ 479us) server_negotiation.cc:299] Negotiation successful
0808 12:57:10.971977 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 12:57:10.973173 32373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:10.985231 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:10.970285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:34367 (local address 127.0.0.1:35652)
0808 12:57:10.984765 (+ 14480us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:10.984979 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14334,"thread_start_us":14039,"threads_started":1}
W20260808 12:57:10.985514 31870 catalog_manager.cc:4584] TS a841737a78a6412fa2e04f7579b0bd04 (127.24.167.1:34367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa111ce453714e21bdf2d0ab0969df36: Network error: Client connection negotiation failed: client connection to 127.24.167.1:34367: connect: Connection refused (error 111)
I20260808 12:57:11.039487 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:11.038915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:34367 (local address 127.0.0.1:35664)
0808 12:57:11.039159 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:11.039298 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260808 12:57:11.040683 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:11.040254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:34367 (local address 127.0.0.1:35680)
0808 12:57:11.040435 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:11.040525 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 12:57:11.094735 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 32191
I20260808 12:57:11.102692 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:11.102082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:34367 (local address 127.0.0.1:35696)
0808 12:57:11.102341 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:11.102492 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260808 12:57:11.103946 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:11.103476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:34367 (local address 127.0.0.1:35698)
0808 12:57:11.103672 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:11.103767 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 12:57:11.120434 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 32385
I20260808 12:57:11.123651 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:11.121269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.24.167.3:43277)
0808 12:57:11.121500 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:11.121506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:11.121529 (+ 23us) server_negotiation.cc:407] Connection header received
0808 12:57:11.121690 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:11.121696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:11.121770 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:11.121884 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:11.122956 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:11.122961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:11.122964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:11.122978 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:11.123010 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:11.123022 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:11.123139 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:11.123277 (+ 138us) server_negotiation.cc:299] Negotiation successful
0808 12:57:11.123322 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 12:57:11.124385 32570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:11.184844 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:11.184300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:34367 (local address 127.0.0.1:35700)
0808 12:57:11.184554 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:11.184658 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260808 12:57:11.186066 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:11.185618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:34367 (local address 127.0.0.1:35708)
0808 12:57:11.185839 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:11.185931 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260808 12:57:11.245456 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 32385
I20260808 12:57:11.272387 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 31840
I20260808 12:57:11.273684 31931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:11.382881 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 31840
I20260808 12:57:11.382975 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:11.382544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:34367 (local address 127.0.0.1:35710)
0808 12:57:11.382721 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:11.382825 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
2026-08-08T12:57:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3934 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 12:57:11.432099 25244 mini_oidc.cc:152] Starting JWKS server
I20260808 12:57:11.435156 25244 webserver.cc:460] Webserver started at https://127.0.0.1:39531/ using document root <none> and password file <none>
I20260808 12:57:11.435529 25244 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 12:57:11.436266 25244 webserver.cc:460] Webserver started at http://127.0.0.1:42257/ using document root <none> and password file <none>
2026-08-08T12:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:57:11Z Disabled control of system clock
I20260808 12:57:11.466972 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:42285
--webserver_interface=127.24.167.62
--webserver_port=0
--builtin_ntp_servers=127.24.167.20:37165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:42285
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39531/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/testchainca.cert with env {}
W20260808 12:57:11.805290 32662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:11.805686 32662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:11.805858 32662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:11.813395 32662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:57:11.813566 32662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:11.813665 32662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:11.813778 32662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:57:11.813867 32662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 12:57:11.813968 32662 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 12:57:11.814049 32662 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39531/jwks.json
I20260808 12:57:11.824337 32662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:37165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:42285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:42285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39531/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:11.825824 32662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:11.827278 32662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:11.840050 32675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:11.840142 32677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:11.840088 32674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:11.843988 32662 server_base.cc:1034] running on GCE node
I20260808 12:57:11.844908 32662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:11.846542 32662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:11.847800 32662 hybrid_clock.cc:648] HybridClock initialized: now 1786193831847762 us; error 46 us; skew 500 ppm
I20260808 12:57:11.848198 32662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:57:11.850807 32662 webserver.cc:460] Webserver started at http://127.24.167.62:41315/ using document root <none> and password file <none>
I20260808 12:57:11.851554 32662 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:11.851699 32662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:11.852041 32662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:11.855166 32662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "7b4943882f714dae9189aeddc4cd7b66"
format_stamp: "Formatted at 2026-08-08 12:57:11 on dist-test-slave-tk9z"
server_key: "56b537013f60f0e51e05297fab3c49df"
server_key_iv: "d31070e500b2960b90a8d38ff54b93bf"
server_key_version: "encryptionkey@0"
I20260808 12:57:11.855991 32662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "7b4943882f714dae9189aeddc4cd7b66"
format_stamp: "Formatted at 2026-08-08 12:57:11 on dist-test-slave-tk9z"
server_key: "56b537013f60f0e51e05297fab3c49df"
server_key_iv: "d31070e500b2960b90a8d38ff54b93bf"
server_key_version: "encryptionkey@0"
I20260808 12:57:11.861366 32662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 12:57:11.865054 32687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:11.866581 32662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 12:57:11.866778 32662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "7b4943882f714dae9189aeddc4cd7b66"
format_stamp: "Formatted at 2026-08-08 12:57:11 on dist-test-slave-tk9z"
server_key: "56b537013f60f0e51e05297fab3c49df"
server_key_iv: "d31070e500b2960b90a8d38ff54b93bf"
server_key_version: "encryptionkey@0"
I20260808 12:57:11.867050 32662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:11.882295 32662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:11.883236 32662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:11.883564 32662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:11.902696 32662 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:42285
I20260808 12:57:11.902715 310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:42285 every 8 connection(s)
I20260808 12:57:11.904125 32662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:57:11.906971 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 32662
I20260808 12:57:11.907176 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:57:11.908039 25244 external_mini_cluster.cc:1428] Setting key 7c9f1d2b154adacf342f0355811663f5
I20260808 12:57:11.908519 311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:11.921438 311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66: Bootstrap starting.
I20260808 12:57:11.922204 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:11.910090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51122 (local address 127.24.167.62:42285)
0808 12:57:11.911185 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:11.911195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:11.911237 (+ 42us) server_negotiation.cc:407] Connection header received
0808 12:57:11.913415 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:11.913436 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:11.913889 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:11.914317 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:11.914961 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:11.916243 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:11.917643 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:11.918343 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:11.918629 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:11.919882 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:11.919890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:11.919903 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:11.920426 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:11.920506 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:11.920527 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:11.920657 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:11.920893 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 12:57:11.921199 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":458,"threads_started":1}
I20260808 12:57:11.925180 311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:11.926270 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 12:57:11.926457 311 log.cc:824] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:11.929291 311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66: No bootstrap required, opened a new log
I20260808 12:57:11.936656 311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4943882f714dae9189aeddc4cd7b66" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42285 } }
I20260808 12:57:11.937076 311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:11.937187 311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4943882f714dae9189aeddc4cd7b66, State: Initialized, Role: FOLLOWER
I20260808 12:57:11.937750 311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4943882f714dae9189aeddc4cd7b66" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42285 } }
I20260808 12:57:11.938014 311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:57:11.938161 311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:57:11.938331 311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:11.940546 311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4943882f714dae9189aeddc4cd7b66" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42285 } }
I20260808 12:57:11.941100 311 leader_election.cc:302] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b4943882f714dae9189aeddc4cd7b66; no voters:
I20260808 12:57:11.941692 311 leader_election.cc:288] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:57:11.941843 317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:11.943100 317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [term 1 LEADER]: Becoming Leader. State: Replica: 7b4943882f714dae9189aeddc4cd7b66, State: Running, Role: LEADER
I20260808 12:57:11.943627 317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4943882f714dae9189aeddc4cd7b66" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42285 } }
I20260808 12:57:11.943998 311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:57:11.946182 318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b4943882f714dae9189aeddc4cd7b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4943882f714dae9189aeddc4cd7b66" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42285 } } }
I20260808 12:57:11.946636 318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:11.946638 321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b4943882f714dae9189aeddc4cd7b66. Latest consensus state: current_term: 1 leader_uuid: "7b4943882f714dae9189aeddc4cd7b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4943882f714dae9189aeddc4cd7b66" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42285 } } }
I20260808 12:57:11.947314 321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:11.949275 328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:57:11.954927 328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:57:11.964751 328 catalog_manager.cc:1347] Generated new cluster ID: 3664ee7c67274566a97940837cfb299d
I20260808 12:57:11.964895 328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:57:11.977306 328 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:57:11.978943 328 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:57:11.986654 328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b4943882f714dae9189aeddc4cd7b66: Generated new TSK 0
I20260808 12:57:11.987548 328 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 12:57:12.034019 353 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 12:57:12.034268 353 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 12:57:12.035662 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:12.009303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.24.167.62:42285)
0808 12:57:12.009545 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:12.009552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:12.010188 (+ 636us) server_negotiation.cc:407] Connection header received
0808 12:57:12.010667 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:12.010672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:12.010757 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:12.010890 (+ 133us) server_negotiation.cc:226] Negotiated authn=JWT
0808 12:57:12.011623 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:12.012396 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:12.014532 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:12.015004 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:12.015110 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:12.015841 (+ 731us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 12:57:12.034774 (+ 18933us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 12:57:12.035257 (+ 483us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":75}
W20260808 12:57:12.035833 314 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 12:57:12.037652 350 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:12.009188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42285 (local address 127.0.0.1:51124)
0808 12:57:12.010054 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:12.010081 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:12.010316 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:12.010962 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:12.010968 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:12.010982 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0808 12:57:12.011454 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:12.011469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:12.012551 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:12.012557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:12.014341 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:12.014396 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:12.015203 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:12.015213 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:12.015417 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:12.015501 (+ 84us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 12:57:12.035117 (+ 19616us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 12:57:12.035570 (+ 453us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.167.62:42285: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":503,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.24.167.62:42285: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 12:57:12.039618 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 32662
I20260808 12:57:12.040907 300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:12.140564 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 32662
2026-08-08T12:57:12Z chronyd exiting
I20260808 12:57:12.195544 25244 test_util.cc:175] -----------------------------------------------
I20260808 12:57:12.195669 25244 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786193784050718-25244-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (768 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 12:57:12.199651 25244 mini_oidc.cc:152] Starting JWKS server
I20260808 12:57:12.201499 25244 webserver.cc:460] Webserver started at https://127.0.0.1:44919/ using document root <none> and password file <none>
I20260808 12:57:12.201751 25244 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 12:57:12.202505 25244 webserver.cc:460] Webserver started at http://127.0.0.1:40505/ using document root <none> and password file <none>
2026-08-08T12:57:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:57:12Z Disabled control of system clock
I20260808 12:57:12.235262 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:42413
--webserver_interface=127.24.167.62
--webserver_port=0
--builtin_ntp_servers=127.24.167.20:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:42413
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44919/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/testchainca.cert with env {}
W20260808 12:57:12.571334 362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:12.571687 362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:12.571857 362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:12.578967 362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:57:12.579108 362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:12.579206 362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:12.579317 362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:57:12.579411 362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 12:57:12.579509 362 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 12:57:12.579591 362 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44919/jwks.json
I20260808 12:57:12.589354 362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:42413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:42413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44919/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:12.590795 362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:12.592115 362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:12.604179 371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:12.604616 377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:12.604650 374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:12.605244 362 server_base.cc:1034] running on GCE node
I20260808 12:57:12.605994 362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:12.607432 362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:12.608664 362 hybrid_clock.cc:648] HybridClock initialized: now 1786193832608620 us; error 30 us; skew 500 ppm
I20260808 12:57:12.609033 362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:57:12.611312 362 webserver.cc:460] Webserver started at http://127.24.167.62:42301/ using document root <none> and password file <none>
I20260808 12:57:12.611972 362 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:12.612066 362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:12.612334 362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:12.615329 362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "3d8b19066f7c4a21b5be77d6c396b603"
format_stamp: "Formatted at 2026-08-08 12:57:12 on dist-test-slave-tk9z"
server_key: "e865cff5116d92f3d03642ff448cc504"
server_key_iv: "37b94fd0583a8770a16716e28be78153"
server_key_version: "encryptionkey@0"
I20260808 12:57:12.616091 362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "3d8b19066f7c4a21b5be77d6c396b603"
format_stamp: "Formatted at 2026-08-08 12:57:12 on dist-test-slave-tk9z"
server_key: "e865cff5116d92f3d03642ff448cc504"
server_key_iv: "37b94fd0583a8770a16716e28be78153"
server_key_version: "encryptionkey@0"
I20260808 12:57:12.621384 362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260808 12:57:12.624998 388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:12.626608 362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:57:12.626827 362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "3d8b19066f7c4a21b5be77d6c396b603"
format_stamp: "Formatted at 2026-08-08 12:57:12 on dist-test-slave-tk9z"
server_key: "e865cff5116d92f3d03642ff448cc504"
server_key_iv: "37b94fd0583a8770a16716e28be78153"
server_key_version: "encryptionkey@0"
I20260808 12:57:12.627089 362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:12.650251 362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:12.651335 362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:12.651686 362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:12.670725 362 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:42413
I20260808 12:57:12.670769 464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:42413 every 8 connection(s)
I20260808 12:57:12.672195 362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:57:12.675801 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 362
I20260808 12:57:12.676076 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:57:12.676753 467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:12.676836 25244 external_mini_cluster.cc:1428] Setting key c24fe5df3b47b8d9fa1c68d56ea6ef2e
I20260808 12:57:12.691370 467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603: Bootstrap starting.
I20260808 12:57:12.691977 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:12.678969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.24.167.62:42413)
0808 12:57:12.680135 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:12.680150 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:12.680198 (+ 48us) server_negotiation.cc:407] Connection header received
0808 12:57:12.682282 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:12.682307 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:12.682930 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:12.683374 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:12.684055 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:12.685487 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:12.687001 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:12.687758 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:12.688038 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:12.689398 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:12.689407 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:12.689422 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:12.689995 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:12.690079 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:12.690102 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:12.690245 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:12.690544 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 12:57:12.690894 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":446,"threads_started":1}
I20260808 12:57:12.695415 467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:12.696378 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 12:57:12.696754 467 log.cc:824] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:12.700095 467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603: No bootstrap required, opened a new log
I20260808 12:57:12.708173 467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8b19066f7c4a21b5be77d6c396b603" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42413 } }
I20260808 12:57:12.708602 467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:12.708742 467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8b19066f7c4a21b5be77d6c396b603, State: Initialized, Role: FOLLOWER
I20260808 12:57:12.709494 467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8b19066f7c4a21b5be77d6c396b603" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42413 } }
I20260808 12:57:12.709861 467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:57:12.710013 467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:57:12.710160 467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:12.712803 467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8b19066f7c4a21b5be77d6c396b603" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42413 } }
I20260808 12:57:12.713522 467 leader_election.cc:302] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d8b19066f7c4a21b5be77d6c396b603; no voters:
I20260808 12:57:12.714318 467 leader_election.cc:288] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:57:12.714421 476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:12.715252 476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [term 1 LEADER]: Becoming Leader. State: Replica: 3d8b19066f7c4a21b5be77d6c396b603, State: Running, Role: LEADER
I20260808 12:57:12.715840 476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8b19066f7c4a21b5be77d6c396b603" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42413 } }
I20260808 12:57:12.716774 467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:57:12.719036 480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d8b19066f7c4a21b5be77d6c396b603. Latest consensus state: current_term: 1 leader_uuid: "3d8b19066f7c4a21b5be77d6c396b603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8b19066f7c4a21b5be77d6c396b603" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42413 } } }
I20260808 12:57:12.719185 477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d8b19066f7c4a21b5be77d6c396b603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8b19066f7c4a21b5be77d6c396b603" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42413 } } }
I20260808 12:57:12.719481 480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:12.719532 477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:12.732299 486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:57:12.739251 486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:57:12.748312 486 catalog_manager.cc:1347] Generated new cluster ID: 3bb29f729fbc40228199e029a898b89e
I20260808 12:57:12.748454 486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:57:12.769964 486 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:57:12.771310 486 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:57:12.780704 486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d8b19066f7c4a21b5be77d6c396b603: Generated new TSK 0
I20260808 12:57:12.781527 486 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 12:57:12.819087 517 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 12:57:12.819234 517 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260808 12:57:12.820623 515 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:12.799813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42413 (local address 127.0.0.1:58192)
0808 12:57:12.800479 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:12.800495 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:12.800612 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:12.801083 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:12.801086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:12.801095 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0808 12:57:12.801362 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:12.801370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:12.802872 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:12.802876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:12.804031 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:12.804042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:12.805045 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:12.805050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:12.805173 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:12.805206 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 12:57:12.820103 (+ 14897us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 12:57:12.820312 (+ 209us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.167.62:42413: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":379,"threads_started":1}
I20260808 12:57:12.820611 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:12.799952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58192 (local address 127.24.167.62:42413)
0808 12:57:12.800152 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:12.800158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:12.800575 (+ 417us) server_negotiation.cc:407] Connection header received
0808 12:57:12.800795 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:12.800801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:12.800905 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:12.801067 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0808 12:57:12.801552 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:12.802737 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:12.804167 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:12.804891 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:12.805028 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:12.805479 (+ 451us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 12:57:12.819848 (+ 14369us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 12:57:12.820282 (+ 434us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58192: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 12:57:12.820762 472 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58192: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44919/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260808 12:57:12.822283 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 362
I20260808 12:57:12.823524 457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:12.921741 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 362
2026-08-08T12:57:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (785 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 12:57:12.985898 25244 mini_oidc.cc:152] Starting JWKS server
I20260808 12:57:12.988060 25244 webserver.cc:460] Webserver started at https://127.0.0.1:34827/ using document root <none> and password file <none>
I20260808 12:57:12.988273 25244 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 12:57:12.988922 25244 webserver.cc:460] Webserver started at http://127.0.0.1:36541/ using document root <none> and password file <none>
2026-08-08T12:57:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:57:13Z Disabled control of system clock
I20260808 12:57:13.022568 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:33315
--webserver_interface=127.24.167.62
--webserver_port=0
--builtin_ntp_servers=127.24.167.20:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:33315
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34827/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 12:57:13.360608 531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:13.360894 531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:13.361086 531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:13.368431 531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:57:13.368563 531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:13.368649 531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:13.368748 531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:57:13.368830 531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 12:57:13.368918 531 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 12:57:13.368974 531 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34827/jwks.json
I20260808 12:57:13.378252 531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:33315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:33315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34827/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:13.379688 531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:13.381538 531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:13.393776 537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:13.393728 541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:13.393687 538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:13.394699 531 server_base.cc:1034] running on GCE node
I20260808 12:57:13.395617 531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:13.397137 531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:13.398425 531 hybrid_clock.cc:648] HybridClock initialized: now 1786193833398335 us; error 74 us; skew 500 ppm
I20260808 12:57:13.398823 531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:57:13.401145 531 webserver.cc:460] Webserver started at http://127.24.167.62:42521/ using document root <none> and password file <none>
I20260808 12:57:13.401885 531 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:13.402029 531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:13.402412 531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:13.405390 531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "30f1951cf5e242a385a24ce61627d832"
format_stamp: "Formatted at 2026-08-08 12:57:13 on dist-test-slave-tk9z"
server_key: "3fc0f89718d5c3385f994caeac81b556"
server_key_iv: "e241dba5527b031f2ca8bee3d0a74a19"
server_key_version: "encryptionkey@0"
I20260808 12:57:13.406216 531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "30f1951cf5e242a385a24ce61627d832"
format_stamp: "Formatted at 2026-08-08 12:57:13 on dist-test-slave-tk9z"
server_key: "3fc0f89718d5c3385f994caeac81b556"
server_key_iv: "e241dba5527b031f2ca8bee3d0a74a19"
server_key_version: "encryptionkey@0"
I20260808 12:57:13.411619 531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 12:57:13.415217 558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:13.416599 531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 12:57:13.416800 531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "30f1951cf5e242a385a24ce61627d832"
format_stamp: "Formatted at 2026-08-08 12:57:13 on dist-test-slave-tk9z"
server_key: "3fc0f89718d5c3385f994caeac81b556"
server_key_iv: "e241dba5527b031f2ca8bee3d0a74a19"
server_key_version: "encryptionkey@0"
I20260808 12:57:13.417042 531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:13.430198 531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:13.431124 531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:13.431459 531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:13.449720 531 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:33315
I20260808 12:57:13.449741 632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:33315 every 8 connection(s)
I20260808 12:57:13.451159 531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:57:13.451983 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 531
I20260808 12:57:13.452200 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:57:13.453027 25244 external_mini_cluster.cc:1428] Setting key 15ead2bd32ffe91275b3668486ab9f7c
I20260808 12:57:13.455883 633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:13.468972 639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:13.455076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.24.167.62:33315)
0808 12:57:13.456160 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:13.456181 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:13.456232 (+ 51us) server_negotiation.cc:407] Connection header received
0808 12:57:13.458569 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:13.458596 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:13.459188 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:13.459630 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:13.460153 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:13.462206 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:13.463666 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:13.464687 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:13.464888 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:13.466031 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:13.466046 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:13.466064 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:13.466842 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:13.466953 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:13.466983 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:13.467108 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:13.467321 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 12:57:13.467687 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":359,"threads_started":1}
I20260808 12:57:13.470592 633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832: Bootstrap starting.
I20260808 12:57:13.473016 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 12:57:13.473945 633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:13.474984 633 log.cc:824] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:13.478504 633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832: No bootstrap required, opened a new log
I20260808 12:57:13.485845 633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f1951cf5e242a385a24ce61627d832" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 33315 } }
I20260808 12:57:13.486236 633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:13.486342 633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f1951cf5e242a385a24ce61627d832, State: Initialized, Role: FOLLOWER
I20260808 12:57:13.486915 633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f1951cf5e242a385a24ce61627d832" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 33315 } }
I20260808 12:57:13.487169 633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:57:13.487304 633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:57:13.487483 633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:13.489563 633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f1951cf5e242a385a24ce61627d832" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 33315 } }
I20260808 12:57:13.490103 633 leader_election.cc:302] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30f1951cf5e242a385a24ce61627d832; no voters:
I20260808 12:57:13.490814 633 leader_election.cc:288] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:57:13.490980 641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:13.491763 641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [term 1 LEADER]: Becoming Leader. State: Replica: 30f1951cf5e242a385a24ce61627d832, State: Running, Role: LEADER
I20260808 12:57:13.492269 641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f1951cf5e242a385a24ce61627d832" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 33315 } }
I20260808 12:57:13.492914 633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:57:13.495111 644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30f1951cf5e242a385a24ce61627d832. Latest consensus state: current_term: 1 leader_uuid: "30f1951cf5e242a385a24ce61627d832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f1951cf5e242a385a24ce61627d832" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 33315 } } }
I20260808 12:57:13.495513 644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:13.496790 642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30f1951cf5e242a385a24ce61627d832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f1951cf5e242a385a24ce61627d832" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 33315 } } }
I20260808 12:57:13.497191 642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:13.498270 653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:57:13.503697 653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:57:13.512475 653 catalog_manager.cc:1347] Generated new cluster ID: 2b7b9db1699c4c3e98604c26677fbe63
I20260808 12:57:13.512627 653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:57:13.534289 653 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:57:13.535646 653 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:57:13.546589 653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30f1951cf5e242a385a24ce61627d832: Generated new TSK 0
I20260808 12:57:13.547631 653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 12:57:13.579901 639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:13.570685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34138 (local address 127.24.167.62:33315)
0808 12:57:13.570934 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:13.570944 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:13.571447 (+ 503us) server_negotiation.cc:407] Connection header received
0808 12:57:13.571722 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:13.571726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:13.571799 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:13.571933 (+ 134us) server_negotiation.cc:226] Negotiated authn=JWT
0808 12:57:13.572536 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:13.573412 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:13.575527 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:13.575992 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:13.576091 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:13.576726 (+ 635us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 12:57:13.579196 (+ 2470us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 12:57:13.579562 (+ 366us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34138: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 12:57:13.580060 639 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34138: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 12:57:13.580476 678 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:13.570650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:33315 (local address 127.0.0.1:34138)
0808 12:57:13.571326 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:13.571348 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:13.571509 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:13.571985 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:13.571990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:13.572004 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0808 12:57:13.572331 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:13.572345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:13.573570 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:13.573576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:13.575349 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:13.575368 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:13.576192 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:13.576200 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:13.576396 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:13.576448 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 12:57:13.579412 (+ 2964us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 12:57:13.579660 (+ 248us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.167.62:33315: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":350,"threads_started":1}
I20260808 12:57:13.582274 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 531
I20260808 12:57:13.583501 624 generic_service.cc:196] Checking for leaks (request via RPC)
E20260808 12:57:13.593063 681 webserver.cc:533] Webserver: sslize failed:
E20260808 12:57:13.593152 681 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 12:57:13.677006 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 531
2026-08-08T12:57:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (754 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T12:57:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:57:13Z Disabled control of system clock
I20260808 12:57:13.771654 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:44149
--webserver_interface=127.24.167.62
--webserver_port=0
--builtin_ntp_servers=127.24.167.20:41323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:44149
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/testchainca.cert with env {}
W20260808 12:57:14.106253 690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:14.106570 690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:14.106658 690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:14.114073 690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:57:14.114194 690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:14.114234 690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:14.114287 690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:57:14.114341 690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 12:57:14.114423 690 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 12:57:14.114461 690 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 12:57:14.124567 690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:41323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:44149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:44149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:14.125890 690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:14.127243 690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:14.139249 698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:14.139289 702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:14.139196 697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:14.140591 690 server_base.cc:1034] running on GCE node
I20260808 12:57:14.141455 690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:14.143067 690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:14.144276 690 hybrid_clock.cc:648] HybridClock initialized: now 1786193834144224 us; error 55 us; skew 500 ppm
I20260808 12:57:14.144639 690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:57:14.147023 690 webserver.cc:460] Webserver started at http://127.24.167.62:35385/ using document root <none> and password file <none>
I20260808 12:57:14.147751 690 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:14.147881 690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:14.148211 690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:14.151305 690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "b2320b922c0040dabb1a6fd75ccfc580"
format_stamp: "Formatted at 2026-08-08 12:57:14 on dist-test-slave-tk9z"
server_key: "008283b1d3f22b01884d5c7574341fcd"
server_key_iv: "318d882ff26e73208423c7394d8e4ecc"
server_key_version: "encryptionkey@0"
I20260808 12:57:14.152132 690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "b2320b922c0040dabb1a6fd75ccfc580"
format_stamp: "Formatted at 2026-08-08 12:57:14 on dist-test-slave-tk9z"
server_key: "008283b1d3f22b01884d5c7574341fcd"
server_key_iv: "318d882ff26e73208423c7394d8e4ecc"
server_key_version: "encryptionkey@0"
I20260808 12:57:14.157389 690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260808 12:57:14.160964 710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:14.162472 690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:57:14.162657 690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "b2320b922c0040dabb1a6fd75ccfc580"
format_stamp: "Formatted at 2026-08-08 12:57:14 on dist-test-slave-tk9z"
server_key: "008283b1d3f22b01884d5c7574341fcd"
server_key_iv: "318d882ff26e73208423c7394d8e4ecc"
server_key_version: "encryptionkey@0"
I20260808 12:57:14.162866 690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:14.180974 690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:14.181872 690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:14.182183 690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:14.200707 690 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:44149
I20260808 12:57:14.200727 779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:44149 every 8 connection(s)
I20260808 12:57:14.202194 690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:57:14.205981 783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:14.212210 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 690
I20260808 12:57:14.212438 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:57:14.213110 25244 external_mini_cluster.cc:1428] Setting key 2aa8a99bf9d8012ba267765f5e1e35e7
I20260808 12:57:14.217532 783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580: Bootstrap starting.
I20260808 12:57:14.221767 783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:14.223052 783 log.cc:824] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:14.226667 783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580: No bootstrap required, opened a new log
I20260808 12:57:14.229624 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:14.215258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.24.167.62:44149)
0808 12:57:14.216364 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:14.216383 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:14.216446 (+ 63us) server_negotiation.cc:407] Connection header received
0808 12:57:14.218914 (+ 2468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:14.218944 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:14.219599 (+ 655us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:14.220059 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:14.220605 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:14.222603 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:14.224317 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:14.225355 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:14.225575 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:14.226759 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:14.226770 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:14.226788 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:57:14.227508 (+ 720us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:14.227619 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:57:14.227645 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:14.227768 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:14.227941 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 12:57:14.228281 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":347,"threads_started":1}
I20260808 12:57:14.233522 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 12:57:14.236104 783 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2320b922c0040dabb1a6fd75ccfc580" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44149 } }
I20260808 12:57:14.236513 783 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:14.236627 783 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2320b922c0040dabb1a6fd75ccfc580, State: Initialized, Role: FOLLOWER
I20260808 12:57:14.237216 783 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2320b922c0040dabb1a6fd75ccfc580" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44149 } }
I20260808 12:57:14.237468 783 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:57:14.237623 783 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:57:14.237797 783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:14.240080 783 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2320b922c0040dabb1a6fd75ccfc580" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44149 } }
I20260808 12:57:14.240635 783 leader_election.cc:302] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2320b922c0040dabb1a6fd75ccfc580; no voters:
I20260808 12:57:14.241261 783 leader_election.cc:288] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:57:14.241463 789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:14.242223 789 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [term 1 LEADER]: Becoming Leader. State: Replica: b2320b922c0040dabb1a6fd75ccfc580, State: Running, Role: LEADER
I20260808 12:57:14.242942 789 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2320b922c0040dabb1a6fd75ccfc580" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44149 } }
I20260808 12:57:14.243631 783 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:57:14.246018 790 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2320b922c0040dabb1a6fd75ccfc580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2320b922c0040dabb1a6fd75ccfc580" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44149 } } }
I20260808 12:57:14.246480 790 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:14.246450 791 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2320b922c0040dabb1a6fd75ccfc580. Latest consensus state: current_term: 1 leader_uuid: "b2320b922c0040dabb1a6fd75ccfc580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2320b922c0040dabb1a6fd75ccfc580" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44149 } } }
I20260808 12:57:14.246783 791 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:14.247967 802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:57:14.253391 802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:57:14.262516 802 catalog_manager.cc:1347] Generated new cluster ID: 6c3409f81a1d4083927e9bf1ac759b1c
I20260808 12:57:14.262633 802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:57:14.274449 802 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:57:14.276211 802 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:57:14.285527 802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2320b922c0040dabb1a6fd75ccfc580: Generated new TSK 0
I20260808 12:57:14.286630 802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 12:57:14.448367 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:14.302130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.24.167.62:44149)
0808 12:57:14.302323 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:14.302330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:14.302728 (+ 398us) server_negotiation.cc:407] Connection header received
0808 12:57:14.302937 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:14.302944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:14.303044 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:14.303230 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0808 12:57:14.303685 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:14.304834 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:14.306275 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:14.306978 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:14.307106 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:14.307619 (+ 513us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 12:57:14.447628 (+140009us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 12:57:14.448021 (+ 393us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57264: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":74}
W20260808 12:57:14.448527 786 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57264: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 12:57:14.448694 831 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:14.301998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:44149 (local address 127.0.0.1:57264)
0808 12:57:14.302619 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:14.302634 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:14.302761 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:14.303255 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:14.303259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:14.303268 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0808 12:57:14.303533 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:14.303541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:14.304988 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:14.304991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:14.306129 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:14.306141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:14.307161 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:14.307165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:14.307286 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:14.307319 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 12:57:14.447969 (+140650us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 12:57:14.448309 (+ 340us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.167.62:44149: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":314,"threads_started":1}
I20260808 12:57:14.450682 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 690
I20260808 12:57:14.451787 773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:14.554452 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 690
2026-08-08T12:57:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (859 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (426 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (421 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:57:15 dist-test-slave-tk9z krb5kdc[854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:57:15 dist-test-slave-tk9z krb5kdc[854](info): set up 2 sockets
Aug 08 12:57:15 dist-test-slave-tk9z krb5kdc[854](info): commencing operation
krb5kdc: starting...
W20260808 12:57:16.698995 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.251s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:57:16 dist-test-slave-tk9z krb5kdc[854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:57:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:57:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 12:57:16.954411 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:35969
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:35969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:17.300381 879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:17.300678 879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:17.300836 879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:17.308238 879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:57:17.308368 879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:17.308458 879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:17.308543 879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:57:17.308612 879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:57:17.318647 879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:35969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:35969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:17.320014 879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:17.321336 879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:17.334276 885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:17.334230 886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:17.334316 889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:17.335089 879 server_base.cc:1034] running on GCE node
I20260808 12:57:17.335868 879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:17.337330 879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:17.338536 879 hybrid_clock.cc:648] HybridClock initialized: now 1786193837338486 us; error 39 us; skew 500 ppm
I20260808 12:57:17.338949 879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:17 dist-test-slave-tk9z krb5kdc[854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193837, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:17.345376 879 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:57:17.347071 879 webserver.cc:460] Webserver started at http://127.24.167.62:43969/ using document root <none> and password file <none>
I20260808 12:57:17.347787 879 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:17.347901 879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:17.348181 879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:17.351436 879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "68309dc1fbd743f184ea0ebeb9340a1b"
format_stamp: "Formatted at 2026-08-08 12:57:17 on dist-test-slave-tk9z"
server_key: "45356430d91ce84a10b0526d4bdc5f03"
server_key_iv: "37409e507031da33252fd684b644a053"
server_key_version: "encryptionkey@0"
I20260808 12:57:17.352240 879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "68309dc1fbd743f184ea0ebeb9340a1b"
format_stamp: "Formatted at 2026-08-08 12:57:17 on dist-test-slave-tk9z"
server_key: "45356430d91ce84a10b0526d4bdc5f03"
server_key_iv: "37409e507031da33252fd684b644a053"
server_key_version: "encryptionkey@0"
I20260808 12:57:17.357769 879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 12:57:17.362193 900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:17.363626 879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 12:57:17.363795 879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "68309dc1fbd743f184ea0ebeb9340a1b"
format_stamp: "Formatted at 2026-08-08 12:57:17 on dist-test-slave-tk9z"
server_key: "45356430d91ce84a10b0526d4bdc5f03"
server_key_iv: "37409e507031da33252fd684b644a053"
server_key_version: "encryptionkey@0"
I20260808 12:57:17.363965 879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:17.387727 879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:17.392030 879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:17.392288 879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:17.411682 879 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:35969
I20260808 12:57:17.411715 972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:35969 every 8 connection(s)
I20260808 12:57:17.413153 879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:57:17.416955 973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:17.418391 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 879
I20260808 12:57:17.418666 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:57:17.419519 25244 external_mini_cluster.cc:1428] Setting key 6f1f4e1af336c2603a9a784761f67529
I20260808 12:57:17.432026 973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b: Bootstrap starting.
Aug 08 12:57:17 dist-test-slave-tk9z krb5kdc[854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:17.435253 973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:17.436611 973 log.cc:824] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:17.439980 973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b: No bootstrap required, opened a new log
I20260808 12:57:17.443765 978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:17.421622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53506 (local address 127.24.167.62:35969)
0808 12:57:17.422442 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:17.422462 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:17.422518 (+ 56us) server_negotiation.cc:407] Connection header received
0808 12:57:17.424217 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:17.424245 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:17.424733 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:17.425179 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:17.426611 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:17.428677 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:17.429986 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:17.431045 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:17.431228 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:17.435391 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:17.435402 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:17.435430 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:17.435475 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:17.439462 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:17.440094 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:17.440102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:17.440109 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:17.440311 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:17.440697 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:17.440703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:17.440706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:17.441352 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:17.441684 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:17.441956 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 12:57:17.442434 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":334,"threads_started":1}
I20260808 12:57:17.449083 973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68309dc1fbd743f184ea0ebeb9340a1b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35969 } }
I20260808 12:57:17.449455 973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:17.449591 973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68309dc1fbd743f184ea0ebeb9340a1b, State: Initialized, Role: FOLLOWER
I20260808 12:57:17.450110 973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68309dc1fbd743f184ea0ebeb9340a1b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35969 } }
I20260808 12:57:17.450423 973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:57:17.450575 973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:57:17.450750 973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:17.452829 973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68309dc1fbd743f184ea0ebeb9340a1b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35969 } }
I20260808 12:57:17.453303 973 leader_election.cc:302] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68309dc1fbd743f184ea0ebeb9340a1b; no voters:
I20260808 12:57:17.453864 973 leader_election.cc:288] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:57:17.454020 981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:17.454795 981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [term 1 LEADER]: Becoming Leader. State: Replica: 68309dc1fbd743f184ea0ebeb9340a1b, State: Running, Role: LEADER
I20260808 12:57:17.455341 981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68309dc1fbd743f184ea0ebeb9340a1b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35969 } }
I20260808 12:57:17.455916 973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:57:17.458319 985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68309dc1fbd743f184ea0ebeb9340a1b. Latest consensus state: current_term: 1 leader_uuid: "68309dc1fbd743f184ea0ebeb9340a1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68309dc1fbd743f184ea0ebeb9340a1b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35969 } } }
I20260808 12:57:17.458535 984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68309dc1fbd743f184ea0ebeb9340a1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68309dc1fbd743f184ea0ebeb9340a1b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35969 } } }
I20260808 12:57:17.458946 985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:17.459050 984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:17.459807 992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:57:17.465720 992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:57:17.473884 992 catalog_manager.cc:1347] Generated new cluster ID: 9fef5f7625d246df98f56494d5a60e45
I20260808 12:57:17.473986 992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:57:17.496390 992 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:57:17.497797 992 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:57:17.506666 992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68309dc1fbd743f184ea0ebeb9340a1b: Generated new TSK 0
I20260808 12:57:17.507689 992 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 12:57:17.579720 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35969
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:17.928860 1010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:17.929158 1010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:17.929332 1010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:17.936628 1010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:17.936744 1010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:17.936939 1010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:57:17.947188 1010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:17.948585 1010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:17.949951 1010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:17.963761 1018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:17.963730 1023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:17.963681 1020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:17.964712 1010 server_base.cc:1034] running on GCE node
I20260808 12:57:17.965329 1010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:17.966300 1010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:17.967542 1010 hybrid_clock.cc:648] HybridClock initialized: now 1786193837967483 us; error 59 us; skew 500 ppm
I20260808 12:57:17.967815 1010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:17 dist-test-slave-tk9z krb5kdc[854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193837, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:17.971217 1010 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 12:57:17.972707 1010 webserver.cc:460] Webserver started at http://127.24.167.1:37909/ using document root <none> and password file <none>
I20260808 12:57:17.973425 1010 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:17.973578 1010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:17.973896 1010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:17.976977 1010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "c57cd7ce4a694d77b2092822d46245ee"
format_stamp: "Formatted at 2026-08-08 12:57:17 on dist-test-slave-tk9z"
server_key: "e384aa21809665a718638837e2c9e902"
server_key_iv: "892368486847c4043e3679bebfa43562"
server_key_version: "encryptionkey@0"
I20260808 12:57:17.977805 1010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "c57cd7ce4a694d77b2092822d46245ee"
format_stamp: "Formatted at 2026-08-08 12:57:17 on dist-test-slave-tk9z"
server_key: "e384aa21809665a718638837e2c9e902"
server_key_iv: "892368486847c4043e3679bebfa43562"
server_key_version: "encryptionkey@0"
I20260808 12:57:17.983079 1010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260808 12:57:17.986698 1031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:17.988006 1010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 12:57:17.988186 1010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "c57cd7ce4a694d77b2092822d46245ee"
format_stamp: "Formatted at 2026-08-08 12:57:17 on dist-test-slave-tk9z"
server_key: "e384aa21809665a718638837e2c9e902"
server_key_iv: "892368486847c4043e3679bebfa43562"
server_key_version: "encryptionkey@0"
I20260808 12:57:17.988405 1010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:18.026650 1010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:18.030766 1010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:18.031086 1010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:18.032132 1010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:18.033447 1010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:18.033560 1010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:18.033708 1010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:18.033782 1010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:18.067299 1010 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:35029
I20260808 12:57:18.067389 1192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:35029 every 8 connection(s)
I20260808 12:57:18.068784 1010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:57:18.078150 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 1010
I20260808 12:57:18.078429 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:57:18.079073 25244 external_mini_cluster.cc:1428] Setting key c9ae800baabc4f8d3249a21dc8e3c328
Aug 08 12:57:18 dist-test-slave-tk9z krb5kdc[854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193837, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:18.089421 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:18.071380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:47763 (local address 127.24.167.62:35969)
0808 12:57:18.071825 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:18.071829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:18.072869 (+ 1040us) server_negotiation.cc:407] Connection header received
0808 12:57:18.075225 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:18.075233 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:18.075329 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:18.075451 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:18.076948 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:18.077753 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:18.079531 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:18.080018 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:18.080175 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:18.083677 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:18.083682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:18.083685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:18.083717 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:18.086342 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:18.087109 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:18.087115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:18.087120 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:18.087247 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:18.087694 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:18.087698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:18.087700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:18.087969 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:18.088145 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:18.088832 (+ 687us) server_negotiation.cc:299] Negotiation successful
0808 12:57:18.089057 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":231,"threads_started":1}
I20260808 12:57:18.090452 1198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:18.071953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:35969 (local address 127.24.167.1:47763)
0808 12:57:18.072688 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:18.072727 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:18.074960 (+ 2233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:18.075696 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:18.075705 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:18.076218 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:18.076776 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:18.076795 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:18.077916 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:18.077920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:18.079383 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:18.079396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:18.080263 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:18.080272 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:18.080538 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:18.081214 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:18.081237 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:18.083460 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:18.086538 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:18.086547 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:18.086562 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:18.086979 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:18.087371 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:18.087376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:18.087380 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:18.087563 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:18.088109 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:18.088119 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:18.088458 (+ 339us) client_negotiation.cc:769] Sending connection context
0808 12:57:18.088715 (+ 257us) client_negotiation.cc:240] Negotiation successful
0808 12:57:18.089222 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":316,"threads_started":1}
I20260808 12:57:18.092049 1193 heartbeater.cc:347] Connected to a master server at 127.24.167.62:35969
I20260808 12:57:18.092413 1193 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:18.093189 1193 heartbeater.cc:511] Master 127.24.167.62:35969 requested a full tablet report, sending...
I20260808 12:57:18.095571 927 ts_manager.cc:194] Registered new tserver with Master: c57cd7ce4a694d77b2092822d46245ee (127.24.167.1:35029)
I20260808 12:57:18.097855 927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:47763
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 12:57:18.129460 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35969
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:18.468237 1203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:18.468526 1203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:18.468714 1203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:18.476028 1203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:18.476135 1203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:18.476328 1203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:57:18.486147 1203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:18.487490 1203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:18.488802 1203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:18.502142 1210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:18.502104 1215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:18.502177 1212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:18.503604 1203 server_base.cc:1034] running on GCE node
I20260808 12:57:18.504208 1203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:18.505139 1203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:18.506418 1203 hybrid_clock.cc:648] HybridClock initialized: now 1786193838506343 us; error 48 us; skew 500 ppm
I20260808 12:57:18.506742 1203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:18 dist-test-slave-tk9z krb5kdc[854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193838, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:18.509876 1203 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 12:57:18.511543 1203 webserver.cc:460] Webserver started at http://127.24.167.2:44373/ using document root <none> and password file <none>
I20260808 12:57:18.512331 1203 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:18.512487 1203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:18.512811 1203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:18.515825 1203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "d0cb6226ed544eb4991ef30bce7665b5"
format_stamp: "Formatted at 2026-08-08 12:57:18 on dist-test-slave-tk9z"
server_key: "935ebb8e51139394bd7061058ae5b753"
server_key_iv: "750ad8cb46d01f3cbd6ea28d409fab58"
server_key_version: "encryptionkey@0"
I20260808 12:57:18.516642 1203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "d0cb6226ed544eb4991ef30bce7665b5"
format_stamp: "Formatted at 2026-08-08 12:57:18 on dist-test-slave-tk9z"
server_key: "935ebb8e51139394bd7061058ae5b753"
server_key_iv: "750ad8cb46d01f3cbd6ea28d409fab58"
server_key_version: "encryptionkey@0"
I20260808 12:57:18.522516 1203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 12:57:18.526127 1227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:18.527522 1203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:57:18.527717 1203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "d0cb6226ed544eb4991ef30bce7665b5"
format_stamp: "Formatted at 2026-08-08 12:57:18 on dist-test-slave-tk9z"
server_key: "935ebb8e51139394bd7061058ae5b753"
server_key_iv: "750ad8cb46d01f3cbd6ea28d409fab58"
server_key_version: "encryptionkey@0"
I20260808 12:57:18.527942 1203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:18.560274 1203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:18.564962 1203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:18.565300 1203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:18.566465 1203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:18.567960 1203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:18.568071 1203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:18.568243 1203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:18.568331 1203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:18.603585 1203 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:32865
I20260808 12:57:18.603632 1387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:32865 every 8 connection(s)
I20260808 12:57:18.604934 1203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:57:18.613636 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 1203
I20260808 12:57:18.613847 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:57:18.614291 25244 external_mini_cluster.cc:1428] Setting key b97491a47b39b9be975a4b2fa0cf9d79
Aug 08 12:57:18 dist-test-slave-tk9z krb5kdc[854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193838, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:18.627099 1394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:18.607678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:33853 (local address 127.24.167.62:35969)
0808 12:57:18.608098 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:18.608102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:18.609070 (+ 968us) server_negotiation.cc:407] Connection header received
0808 12:57:18.611143 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:18.611149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:18.611234 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:18.611369 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:18.613005 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:18.613770 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:18.615471 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:18.615908 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:18.616077 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:18.619893 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:18.619902 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:18.619910 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:18.619955 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:18.623572 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:18.624681 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:18.624690 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:18.624696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:18.624877 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:18.625306 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:18.625312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:18.625318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:18.625714 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:18.625875 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:18.626662 (+ 787us) server_negotiation.cc:299] Negotiation successful
0808 12:57:18.626832 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":204,"threads_started":1}
I20260808 12:57:18.628633 1395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:18.608215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:35969 (local address 127.24.167.2:33853)
0808 12:57:18.608911 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:18.608949 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:18.610871 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:18.611606 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:18.611614 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:18.612241 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:18.612820 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:18.612838 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:18.613958 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:18.613963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:18.615301 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:18.615318 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:18.616231 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:18.616239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:18.616511 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:18.617224 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:18.617247 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:18.619654 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:18.623732 (+ 4078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:18.623742 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:18.623759 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:18.624536 (+ 777us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:18.624993 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:18.624996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:18.624998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:18.625182 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:18.625839 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:18.625844 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:18.626218 (+ 374us) client_negotiation.cc:769] Sending connection context
0808 12:57:18.626557 (+ 339us) client_negotiation.cc:240] Negotiation successful
0808 12:57:18.627149 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":275,"threads_started":1}
I20260808 12:57:18.630210 1389 heartbeater.cc:347] Connected to a master server at 127.24.167.62:35969
I20260808 12:57:18.630581 1389 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:18.631319 1389 heartbeater.cc:511] Master 127.24.167.62:35969 requested a full tablet report, sending...
I20260808 12:57:18.632864 927 ts_manager.cc:194] Registered new tserver with Master: d0cb6226ed544eb4991ef30bce7665b5 (127.24.167.2:32865)
I20260808 12:57:18.633724 927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:33853
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 12:57:18.669502 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35969
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:19.012145 1398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:19.012501 1398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:19.012698 1398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:19.019989 1398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:19.020094 1398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:19.020208 1398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:57:19.029803 1398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:19.031132 1398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:19.032538 1398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:19.045950 1411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:19.045872 1405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:19.045919 1406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:19.047268 1398 server_base.cc:1034] running on GCE node
I20260808 12:57:19.047866 1398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:19.048755 1398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:19.049947 1398 hybrid_clock.cc:648] HybridClock initialized: now 1786193839049888 us; error 57 us; skew 500 ppm
I20260808 12:57:19.050226 1398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193839, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:19.053586 1398 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 12:57:19.055076 1398 webserver.cc:460] Webserver started at http://127.24.167.3:36493/ using document root <none> and password file <none>
I20260808 12:57:19.055863 1398 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:19.056023 1398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:19.056355 1398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:19.059520 1398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "f238c66e9aa44cb8913ce0dbedd45409"
format_stamp: "Formatted at 2026-08-08 12:57:19 on dist-test-slave-tk9z"
server_key: "e2bf7dc9b80d4f5ec37483fd6ef2d83c"
server_key_iv: "3a70d27a8f532b0d316b24793c426860"
server_key_version: "encryptionkey@0"
I20260808 12:57:19.060349 1398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "f238c66e9aa44cb8913ce0dbedd45409"
format_stamp: "Formatted at 2026-08-08 12:57:19 on dist-test-slave-tk9z"
server_key: "e2bf7dc9b80d4f5ec37483fd6ef2d83c"
server_key_iv: "3a70d27a8f532b0d316b24793c426860"
server_key_version: "encryptionkey@0"
I20260808 12:57:19.065392 1398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260808 12:57:19.069020 1422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:19.070432 1398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 12:57:19.070622 1398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "f238c66e9aa44cb8913ce0dbedd45409"
format_stamp: "Formatted at 2026-08-08 12:57:19 on dist-test-slave-tk9z"
server_key: "e2bf7dc9b80d4f5ec37483fd6ef2d83c"
server_key_iv: "3a70d27a8f532b0d316b24793c426860"
server_key_version: "encryptionkey@0"
I20260808 12:57:19.070847 1398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:19.082463 1398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:19.086726 1398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:19.087026 1398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:19.087980 1398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:19.089323 1398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:19.089437 1398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:19.089588 1398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:19.089663 1398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:19.101012 1193 heartbeater.cc:503] Master 127.24.167.62:35969 was elected leader, sending a full tablet report...
I20260808 12:57:19.126484 1398 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:40431
I20260808 12:57:19.126576 1603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:40431 every 8 connection(s)
I20260808 12:57:19.127817 1398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:57:19.133018 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 1398
I20260808 12:57:19.133255 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:57:19.133710 25244 external_mini_cluster.cc:1428] Setting key c89557e392276574e95ea9d744d8f216
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193839, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:19.149426 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:19.130693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:47393 (local address 127.24.167.62:35969)
0808 12:57:19.131145 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:19.131150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:19.132283 (+ 1133us) server_negotiation.cc:407] Connection header received
0808 12:57:19.134508 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:19.134514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:19.134624 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:19.134752 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:19.136843 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.137687 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.139317 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.139715 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.139808 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:19.143234 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:19.143239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:19.143242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:19.143274 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:19.145943 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:19.146755 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:19.146760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:19.146762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:19.146885 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:19.147459 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:19.147464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:19.147467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:19.147805 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:19.147934 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:19.148825 (+ 891us) server_negotiation.cc:299] Negotiation successful
0808 12:57:19.149003 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":224,"threads_started":1}
I20260808 12:57:19.150457 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:19.131250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:35969 (local address 127.24.167.3:47393)
0808 12:57:19.132114 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:19.132169 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:19.134147 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:19.135036 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:19.135048 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:19.135774 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:19.136651 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:19.136677 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.137830 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.137834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:19.139188 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:19.139205 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.139938 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.139943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:19.140146 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:19.140783 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:19.140806 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:19.143017 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:19.146088 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:19.146097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:19.146113 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:19.146622 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:19.146999 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:19.147004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:19.147008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:19.147291 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:19.147941 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:19.147950 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:19.148346 (+ 396us) client_negotiation.cc:769] Sending connection context
0808 12:57:19.148677 (+ 331us) client_negotiation.cc:240] Negotiation successful
0808 12:57:19.149212 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":352,"threads_started":1}
I20260808 12:57:19.151744 1607 heartbeater.cc:347] Connected to a master server at 127.24.167.62:35969
I20260808 12:57:19.152056 1607 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:19.152753 1607 heartbeater.cc:511] Master 127.24.167.62:35969 requested a full tablet report, sending...
I20260808 12:57:19.154124 927 ts_manager.cc:194] Registered new tserver with Master: f238c66e9aa44cb8913ce0dbedd45409 (127.24.167.3:40431)
I20260808 12:57:19.154922 927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:47393
I20260808 12:57:19.161384 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/test-admin.keytab.
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:19.192176 25244 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 12:57:19.197563 1619 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:19.194868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.24.167.1:35029)
0808 12:57:19.195628 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:19.195635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:19.195663 (+ 28us) server_negotiation.cc:407] Connection header received
0808 12:57:19.195727 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:19.195733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:19.195960 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:19.196088 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:19.197110 (+ 1022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60758: BlockingRecv error: recv got EOF from 127.0.0.1:60758 (error 108)
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":455,"threads_started":1}
W20260808 12:57:19.198069 25244 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/krb5cc' of type FILE
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:19.199685 25244 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:19.201109 25244 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:57:19.219664 1619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:19.202663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60764 (local address 127.24.167.1:35029)
0808 12:57:19.202813 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:19.202816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:19.203173 (+ 357us) server_negotiation.cc:407] Connection header received
0808 12:57:19.203311 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:19.203314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:19.203368 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:19.203471 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:19.204430 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.205410 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.206692 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.207230 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.207334 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:19.214144 (+ 6810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:19.214159 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:19.214176 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:19.214214 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:19.217211 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:19.217824 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:19.217832 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:19.217834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:19.217969 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:19.218282 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:19.218285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:19.218286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:19.218719 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:19.218962 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:19.219193 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 12:57:19.219397 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 12:57:19.225473 1619 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:19.223778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60770 (local address 127.24.167.1:35029)
0808 12:57:19.223993 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:19.223999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:19.224166 (+ 167us) server_negotiation.cc:407] Connection header received
0808 12:57:19.224285 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:19.224289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:19.224343 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:19.224418 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:19.225198 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60770: BlockingRecv error: recv got EOF from 127.0.0.1:60770 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260808 12:57:19.225577 25244 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/krb5cc' of type FILE
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:19.227159 25244 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:19.228436 25244 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:57:19.242564 1619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:19.229748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.24.167.1:35029)
0808 12:57:19.229922 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:19.229928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:19.230088 (+ 160us) server_negotiation.cc:407] Connection header received
0808 12:57:19.230202 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:19.230206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:19.230255 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:19.230339 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:19.231309 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.232011 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.233352 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.233800 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.233938 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:19.237455 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:19.237460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:19.237463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:19.237495 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:19.239976 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:19.240616 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:19.240623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:19.240626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:19.240759 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:19.241150 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:19.241155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:19.241159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:19.241429 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:19.241725 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:19.242003 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 12:57:19.242207 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260808 12:57:19.246846 1619 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:19.245307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.24.167.1:35029)
0808 12:57:19.245501 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:19.245507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:19.245631 (+ 124us) server_negotiation.cc:407] Connection header received
0808 12:57:19.245751 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:19.245754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:19.245820 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:19.245896 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:19.246608 (+ 712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60800: BlockingRecv error: recv got EOF from 127.0.0.1:60800 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260808 12:57:19.247400 25244 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786193784050718-25244-0/krb5kdc/krb5cc)
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:19.248903 25244 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:19.250223 25244 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:57:19.264281 1619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:19.251557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60806 (local address 127.24.167.1:35029)
0808 12:57:19.251764 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:19.251769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:19.252029 (+ 260us) server_negotiation.cc:407] Connection header received
0808 12:57:19.252204 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:19.252211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:19.252270 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:19.252369 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:19.253312 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.254039 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.255310 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.255722 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.255820 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:19.259366 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:19.259370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:19.259374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:19.259404 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:19.261940 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:19.262591 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:19.262595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:19.262597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:19.262722 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:19.263055 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:19.263058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:19.263060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:19.263325 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:19.263576 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:19.263761 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 12:57:19.263977 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 12:57:19.265810 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 1010
I20260808 12:57:19.277161 1619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:19.266560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60822 (local address 127.24.167.1:35029)
0808 12:57:19.266700 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:19.266704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:19.267038 (+ 334us) server_negotiation.cc:407] Connection header received
0808 12:57:19.267143 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:19.267146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:19.267195 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:19.267267 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:19.268212 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.268907 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.270023 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.270520 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.270611 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:19.272516 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:19.272520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:19.272523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:19.272550 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:19.274921 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:19.275585 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:19.275589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:19.275591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:19.275714 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:19.276014 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:19.276017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:19.276019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:19.276241 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:19.276431 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:19.276660 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 12:57:19.276837 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 12:57:19.277792 1187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:19.361272 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 1010
I20260808 12:57:19.386664 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 1203
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 12:57:19.403060 1634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:19.387758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.24.167.2:32865)
0808 12:57:19.388287 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:19.388294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:19.388320 (+ 26us) server_negotiation.cc:407] Connection header received
0808 12:57:19.388418 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:19.388425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:19.388633 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:19.388825 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:19.389867 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.390999 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.392282 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.392834 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.392979 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:19.396899 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:19.396911 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:19.396925 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:19.396960 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:19.400018 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:19.400764 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:19.400772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:19.400774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:19.400906 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:19.401313 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:19.401319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:19.401322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:19.401699 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:19.402035 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:19.402425 (+ 390us) server_negotiation.cc:299] Negotiation successful
0808 12:57:19.402710 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":261,"threads_started":1}
I20260808 12:57:19.404501 1381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:19.508518 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 1203
I20260808 12:57:19.531597 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 1398
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 12:57:19.547765 1638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:19.532697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.24.167.3:40431)
0808 12:57:19.533346 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:19.533354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:19.533385 (+ 31us) server_negotiation.cc:407] Connection header received
0808 12:57:19.533447 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:19.533456 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:19.533669 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:19.533843 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:19.535087 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.536090 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.537366 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:19.537924 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:19.538091 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:19.542678 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:19.542689 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:19.542703 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:19.542737 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:19.545312 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:19.545911 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:19.545918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:19.545920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:19.546037 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:19.546418 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:19.546440 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:19.546446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:19.546802 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:19.547046 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:19.547273 (+ 227us) server_negotiation.cc:299] Negotiation successful
0808 12:57:19.547482 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":331,"threads_started":1}
I20260808 12:57:19.549046 1599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:19.631542 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 1398
I20260808 12:57:19.655635 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 879
I20260808 12:57:19.656867 966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:19.762410 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 879
2026-08-08T12:57:19Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4366 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[1644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[1644](info): set up 2 sockets
Aug 08 12:57:19 dist-test-slave-tk9z krb5kdc[1644](info): commencing operation
krb5kdc: starting...
W20260808 12:57:21.083945 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:57:21 dist-test-slave-tk9z krb5kdc[1644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:57:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:57:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.62.keytab.
Entry for principal oryx/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.62.keytab.
I20260808 12:57:21.310811 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:34517
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.62.keytab
--principal=oryx/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:34517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:21.648095 1660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:21.648386 1660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:21.648586 1660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:21.656032 1660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:57:21.656149 1660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:21.656235 1660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:21.656318 1660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:57:21.656383 1660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:57:21.666536 1660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:34517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.62.keytab
--principal=oryx/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:34517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:21.667933 1660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:21.669273 1660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:21.681383 1670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:21.681526 1672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:21.681546 1668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:21.682538 1660 server_base.cc:1034] running on GCE node
I20260808 12:57:21.683288 1660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:21.684676 1660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:21.685918 1660 hybrid_clock.cc:648] HybridClock initialized: now 1786193841685882 us; error 46 us; skew 500 ppm
I20260808 12:57:21.686273 1660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:21 dist-test-slave-tk9z krb5kdc[1644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193841, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:21.689503 1660 init.cc:377] Logged in from keytab as oryx/127.24.167.62@KRBTEST.COM (short username oryx)
I20260808 12:57:21.690949 1660 webserver.cc:460] Webserver started at http://127.24.167.62:36927/ using document root <none> and password file <none>
I20260808 12:57:21.691642 1660 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:21.691747 1660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:21.692003 1660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:21.694933 1660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "e2cb52625c1c4008a860a8dc90be27e5"
format_stamp: "Formatted at 2026-08-08 12:57:21 on dist-test-slave-tk9z"
server_key: "99d4aacfaabaf194dc9adcd2bd3097cf"
server_key_iv: "767cf0620002dc4e4491d763588ef7ab"
server_key_version: "encryptionkey@0"
I20260808 12:57:21.695637 1660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "e2cb52625c1c4008a860a8dc90be27e5"
format_stamp: "Formatted at 2026-08-08 12:57:21 on dist-test-slave-tk9z"
server_key: "99d4aacfaabaf194dc9adcd2bd3097cf"
server_key_iv: "767cf0620002dc4e4491d763588ef7ab"
server_key_version: "encryptionkey@0"
I20260808 12:57:21.700556 1660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 12:57:21.703881 1685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:21.705155 1660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 12:57:21.705335 1660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "e2cb52625c1c4008a860a8dc90be27e5"
format_stamp: "Formatted at 2026-08-08 12:57:21 on dist-test-slave-tk9z"
server_key: "99d4aacfaabaf194dc9adcd2bd3097cf"
server_key_iv: "767cf0620002dc4e4491d763588ef7ab"
server_key_version: "encryptionkey@0"
I20260808 12:57:21.705530 1660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:21.741346 1660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:21.746105 1660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:21.746537 1660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:21.765980 1660 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:34517
I20260808 12:57:21.766021 1779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:34517 every 8 connection(s)
I20260808 12:57:21.767477 1660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:57:21.771391 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:21.773342 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 1660
I20260808 12:57:21.773582 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:57:21.774286 25244 external_mini_cluster.cc:1428] Setting key b3fe80e58090dbbef6b0f6f8971abde5
I20260808 12:57:21.785084 1781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5: Bootstrap starting.
I20260808 12:57:21.788507 1781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:21.789880 1781 log.cc:824] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5: Log is configured to *not* fsync() on all Append() calls
Aug 08 12:57:21 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
I20260808 12:57:21.793514 1781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5: No bootstrap required, opened a new log
I20260808 12:57:21.804533 1785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:21.776619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.24.167.62:34517)
0808 12:57:21.777510 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:21.777528 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:21.777582 (+ 54us) server_negotiation.cc:407] Connection header received
0808 12:57:21.779986 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:21.780017 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:21.780532 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:21.780993 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:21.782640 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:21.784612 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:21.787602 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:21.788683 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:21.788888 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:21.794615 (+ 5727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:21.794628 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:21.794653 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:21.794700 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:21.799161 (+ 4461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:21.800164 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:21.800173 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:21.800182 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:21.800412 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:21.800887 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:21.800892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:21.800896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:21.801700 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:21.802044 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:21.802564 (+ 520us) server_negotiation.cc:299] Negotiation successful
0808 12:57:21.803074 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":389,"threads_started":1}
I20260808 12:57:21.804553 1781 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } }
I20260808 12:57:21.805094 1781 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:21.805228 1781 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2cb52625c1c4008a860a8dc90be27e5, State: Initialized, Role: FOLLOWER
I20260808 12:57:21.805884 1781 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } }
I20260808 12:57:21.806157 1781 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:57:21.806314 1781 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:57:21.806571 1781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:21.809216 1781 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } }
I20260808 12:57:21.809844 1781 leader_election.cc:302] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2cb52625c1c4008a860a8dc90be27e5; no voters:
I20260808 12:57:21.810673 1781 leader_election.cc:288] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:57:21.810807 1787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:21.812860 1781 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:57:21.814661 1787 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [term 1 LEADER]: Becoming Leader. State: Replica: e2cb52625c1c4008a860a8dc90be27e5, State: Running, Role: LEADER
I20260808 12:57:21.815371 1787 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } }
I20260808 12:57:21.818432 1788 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } }
I20260808 12:57:21.819010 1788 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:21.819921 1795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:57:21.819784 1789 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2cb52625c1c4008a860a8dc90be27e5. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } }
I20260808 12:57:21.820230 1789 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:21.837667 1795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:57:21.858498 1795 catalog_manager.cc:1347] Generated new cluster ID: ffe882dca60648c4b97a045851f266de
I20260808 12:57:21.858641 1795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:57:21.871031 1795 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:57:21.872447 1795 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:57:21.886127 1795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5: Generated new TSK 0
I20260808 12:57:21.887049 1795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.1.keytab.
Entry for principal oryx/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.1.keytab.
I20260808 12:57:21.958158 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:34517
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.1.keytab
--principal=oryx/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:22.305505 1815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:22.305774 1815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:22.305990 1815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:22.313383 1815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:22.313508 1815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:22.313715 1815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:57:22.323549 1815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.1.keytab
--principal=oryx/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:34517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:22.324838 1815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:22.326136 1815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:22.339115 1822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:22.339080 1824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:22.339078 1826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:22.340284 1815 server_base.cc:1034] running on GCE node
I20260808 12:57:22.340869 1815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:22.341764 1815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:22.342995 1815 hybrid_clock.cc:648] HybridClock initialized: now 1786193842342951 us; error 51 us; skew 500 ppm
I20260808 12:57:22.343307 1815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:22 dist-test-slave-tk9z krb5kdc[1644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193842, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:22.350656 1815 init.cc:377] Logged in from keytab as oryx/127.24.167.1@KRBTEST.COM (short username oryx)
I20260808 12:57:22.352049 1815 webserver.cc:460] Webserver started at http://127.24.167.1:35375/ using document root <none> and password file <none>
I20260808 12:57:22.352793 1815 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:22.352945 1815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:22.353281 1815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:22.356215 1815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "bc076c248e2541029f9460a5295055b9"
format_stamp: "Formatted at 2026-08-08 12:57:22 on dist-test-slave-tk9z"
server_key: "75adc5363817b8e437c567d9052fe088"
server_key_iv: "fa31ceb18fd623500f12bec6a2a96a55"
server_key_version: "encryptionkey@0"
I20260808 12:57:22.357014 1815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "bc076c248e2541029f9460a5295055b9"
format_stamp: "Formatted at 2026-08-08 12:57:22 on dist-test-slave-tk9z"
server_key: "75adc5363817b8e437c567d9052fe088"
server_key_iv: "fa31ceb18fd623500f12bec6a2a96a55"
server_key_version: "encryptionkey@0"
I20260808 12:57:22.363013 1815 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260808 12:57:22.366226 1837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:22.367527 1815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:57:22.367708 1815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "bc076c248e2541029f9460a5295055b9"
format_stamp: "Formatted at 2026-08-08 12:57:22 on dist-test-slave-tk9z"
server_key: "75adc5363817b8e437c567d9052fe088"
server_key_iv: "fa31ceb18fd623500f12bec6a2a96a55"
server_key_version: "encryptionkey@0"
I20260808 12:57:22.367908 1815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:22.410494 1815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:22.415365 1815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:22.415699 1815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:22.416745 1815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:22.418147 1815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:22.418260 1815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:22.418444 1815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:22.418524 1815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260808 12:57:22.451872 1815 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:32969
I20260808 12:57:22.451932 1996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:32969 every 8 connection(s)
I20260808 12:57:22.453188 1815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:57:22.454658 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 1815
I20260808 12:57:22.454916 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:57:22.455673 25244 external_mini_cluster.cc:1428] Setting key 5f87ef1c123d92ce1def4df32f05caa2
Aug 08 12:57:22 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193842, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
I20260808 12:57:22.474233 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:22.456124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:54523 (local address 127.24.167.62:34517)
0808 12:57:22.456859 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:22.456867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:22.457836 (+ 969us) server_negotiation.cc:407] Connection header received
0808 12:57:22.459777 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:22.459790 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:22.459877 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:22.460010 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:22.461535 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:22.462261 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:22.463837 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:22.464247 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:22.464394 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:22.468189 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:22.468194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:22.468198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:22.468230 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:22.470952 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:22.471807 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:22.471815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:22.471823 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:22.471944 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:22.472488 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:22.472492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:22.472494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:22.472766 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:22.472943 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:22.473637 (+ 694us) server_negotiation.cc:299] Negotiation successful
0808 12:57:22.473861 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":464,"threads_started":1}
I20260808 12:57:22.475208 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:22.456762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.1:54523)
0808 12:57:22.457669 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:22.457723 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:22.459499 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:22.460246 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:22.460254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:22.460771 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:22.461351 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:22.461368 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:22.462447 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:22.462451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:22.463659 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:22.463671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:22.464481 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:22.464490 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:22.464736 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:22.465443 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:22.465464 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:22.467941 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:22.471124 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:22.471133 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:22.471167 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:22.471678 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:22.472080 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:22.472087 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:22.472090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:22.472357 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:22.472909 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:22.472919 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:22.473262 (+ 343us) client_negotiation.cc:769] Sending connection context
0808 12:57:22.473520 (+ 258us) client_negotiation.cc:240] Negotiation successful
0808 12:57:22.474056 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":335,"threads_started":1}
I20260808 12:57:22.476840 1997 heartbeater.cc:347] Connected to a master server at 127.24.167.62:34517
I20260808 12:57:22.477172 1997 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:22.477931 1997 heartbeater.cc:511] Master 127.24.167.62:34517 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.24.167.2@KRBTEST.COM; defaulting to no policy
I20260808 12:57:22.480927 1716 ts_manager.cc:194] Registered new tserver with Master: bc076c248e2541029f9460a5295055b9 (127.24.167.1:32969)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.167.2@KRBTEST.COM" created.
I20260808 12:57:22.482599 1716 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:54523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.2.keytab.
Entry for principal oryx/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.2.keytab.
I20260808 12:57:22.507822 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:34517
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.2.keytab
--principal=oryx/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:22.859567 2006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:22.859881 2006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:22.859988 2006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:22.868090 2006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:22.868214 2006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:22.868335 2006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:57:22.878525 2006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.2.keytab
--principal=oryx/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:34517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:22.879921 2006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:22.881201 2006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:22.894477 2017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:22.894534 2012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:22.894506 2014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:22.895638 2006 server_base.cc:1034] running on GCE node
I20260808 12:57:22.896282 2006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:22.897251 2006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:22.898496 2006 hybrid_clock.cc:648] HybridClock initialized: now 1786193842898409 us; error 56 us; skew 500 ppm
I20260808 12:57:22.898823 2006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:22 dist-test-slave-tk9z krb5kdc[1644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193842, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:22.901948 2006 init.cc:377] Logged in from keytab as oryx/127.24.167.2@KRBTEST.COM (short username oryx)
I20260808 12:57:22.903400 2006 webserver.cc:460] Webserver started at http://127.24.167.2:34823/ using document root <none> and password file <none>
I20260808 12:57:22.904106 2006 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:22.904255 2006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:22.904568 2006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:22.908015 2006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "927f604da71144bd8c483bee804ca845"
format_stamp: "Formatted at 2026-08-08 12:57:22 on dist-test-slave-tk9z"
server_key: "55219ed598cf97bf7d45b1c5d3d22e2b"
server_key_iv: "1832c3913eefaa2c239e751122d394bf"
server_key_version: "encryptionkey@0"
I20260808 12:57:22.909170 2006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "927f604da71144bd8c483bee804ca845"
format_stamp: "Formatted at 2026-08-08 12:57:22 on dist-test-slave-tk9z"
server_key: "55219ed598cf97bf7d45b1c5d3d22e2b"
server_key_iv: "1832c3913eefaa2c239e751122d394bf"
server_key_version: "encryptionkey@0"
I20260808 12:57:22.916095 2006 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260808 12:57:22.919445 2027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:22.920797 2006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 12:57:22.921057 2006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "927f604da71144bd8c483bee804ca845"
format_stamp: "Formatted at 2026-08-08 12:57:22 on dist-test-slave-tk9z"
server_key: "55219ed598cf97bf7d45b1c5d3d22e2b"
server_key_iv: "1832c3913eefaa2c239e751122d394bf"
server_key_version: "encryptionkey@0"
I20260808 12:57:22.921247 2006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:22.938194 2006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:22.942124 2006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:22.942452 2006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:22.943385 2006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:22.944733 2006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:22.944809 2006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:22.944896 2006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:22.944933 2006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:22.978649 2006 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:34171
I20260808 12:57:22.978695 2187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:34171 every 8 connection(s)
I20260808 12:57:22.979976 2006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:57:22.981503 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 2006
I20260808 12:57:22.981745 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:57:22.982254 25244 external_mini_cluster.cc:1428] Setting key 7f0bb4ffb2e5bd95576f9beff9f80401
Aug 08 12:57:22 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193842, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
I20260808 12:57:23.002631 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:22.982998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:44609 (local address 127.24.167.62:34517)
0808 12:57:22.983690 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:22.983697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:22.984850 (+ 1153us) server_negotiation.cc:407] Connection header received
0808 12:57:22.987790 (+ 2940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:22.987794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:22.987860 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:22.987952 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:22.990010 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:22.990844 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:22.992810 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:22.993250 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:22.993385 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:22.996834 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:22.996839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:22.996842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:22.996875 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:22.999537 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:23.000265 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:23.000277 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:23.000280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:23.000408 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:23.000837 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:23.000844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:23.000847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:23.001120 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:23.001280 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:23.001972 (+ 692us) server_negotiation.cc:299] Negotiation successful
0808 12:57:23.002207 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":459,"threads_started":1}
I20260808 12:57:23.003685 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:22.983289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.2:44609)
0808 12:57:22.984665 (+ 1376us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:22.984715 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:22.987475 (+ 2760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:22.988233 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:22.988247 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:22.989037 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:22.989840 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:22.989867 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:22.991050 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:22.991056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:22.992655 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:22.992669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:22.993498 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:22.993506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:22.993734 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:22.994472 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:22.994496 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:22.996605 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:22.999681 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:22.999693 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:22.999708 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:23.000135 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:23.000522 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:23.000527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:23.000531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:23.000725 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:23.001251 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:23.001259 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:23.001610 (+ 351us) client_negotiation.cc:769] Sending connection context
0808 12:57:23.001884 (+ 274us) client_negotiation.cc:240] Negotiation successful
0808 12:57:23.002481 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"thread_start_us":780,"threads_started":1}
I20260808 12:57:23.005240 2190 heartbeater.cc:347] Connected to a master server at 127.24.167.62:34517
I20260808 12:57:23.005561 2190 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:23.006346 2190 heartbeater.cc:511] Master 127.24.167.62:34517 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.24.167.3@KRBTEST.COM; defaulting to no policy
I20260808 12:57:23.007900 1716 ts_manager.cc:194] Registered new tserver with Master: 927f604da71144bd8c483bee804ca845 (127.24.167.2:34171)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.167.3@KRBTEST.COM" created.
I20260808 12:57:23.009150 1716 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:44609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.3.keytab.
Entry for principal oryx/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.3.keytab.
I20260808 12:57:23.037050 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:34517
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.3.keytab
--principal=oryx/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:23.375373 2199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:23.375662 2199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:23.375758 2199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:23.384161 2199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:23.384270 2199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:23.384384 2199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:57:23.394271 2199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.3.keytab
--principal=oryx/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:34517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:23.395627 2199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:23.396925 2199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:23.410662 2211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:23.410755 2214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:23.411185 2210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:23.411576 2199 server_base.cc:1034] running on GCE node
I20260808 12:57:23.412134 2199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:23.413036 2199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:23.414219 2199 hybrid_clock.cc:648] HybridClock initialized: now 1786193843414123 us; error 85 us; skew 500 ppm
I20260808 12:57:23.414618 2199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:23 dist-test-slave-tk9z krb5kdc[1644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193843, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:23.417655 2199 init.cc:377] Logged in from keytab as oryx/127.24.167.3@KRBTEST.COM (short username oryx)
I20260808 12:57:23.419225 2199 webserver.cc:460] Webserver started at http://127.24.167.3:40679/ using document root <none> and password file <none>
I20260808 12:57:23.419919 2199 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:23.420070 2199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:23.420374 2199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:23.423491 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "62ff2f69928b42d3b99721fa260c0c5d"
format_stamp: "Formatted at 2026-08-08 12:57:23 on dist-test-slave-tk9z"
server_key: "7639de76856eca476791dfc438474fec"
server_key_iv: "398b9fbff32757653061e45a59f63b1a"
server_key_version: "encryptionkey@0"
I20260808 12:57:23.424285 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "62ff2f69928b42d3b99721fa260c0c5d"
format_stamp: "Formatted at 2026-08-08 12:57:23 on dist-test-slave-tk9z"
server_key: "7639de76856eca476791dfc438474fec"
server_key_iv: "398b9fbff32757653061e45a59f63b1a"
server_key_version: "encryptionkey@0"
I20260808 12:57:23.429594 2199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 12:57:23.432981 2226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:23.434245 2199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 12:57:23.434480 2199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "62ff2f69928b42d3b99721fa260c0c5d"
format_stamp: "Formatted at 2026-08-08 12:57:23 on dist-test-slave-tk9z"
server_key: "7639de76856eca476791dfc438474fec"
server_key_iv: "398b9fbff32757653061e45a59f63b1a"
server_key_version: "encryptionkey@0"
I20260808 12:57:23.434684 2199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:23.455381 2199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:23.459254 2199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:23.459553 2199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:23.460510 2199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:23.461845 2199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:23.461956 2199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:23.462107 2199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:23.462181 2199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:23.485488 1997 heartbeater.cc:503] Master 127.24.167.62:34517 was elected leader, sending a full tablet report...
I20260808 12:57:23.496805 2199 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:39209
I20260808 12:57:23.496889 2384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:39209 every 8 connection(s)
I20260808 12:57:23.498117 2199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:57:23.499903 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 2199
I20260808 12:57:23.500133 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:57:23.500633 25244 external_mini_cluster.cc:1428] Setting key 5c13f45caf44e06d4dbbf5ee126d65c6
Aug 08 12:57:23 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193843, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
I20260808 12:57:23.519191 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:23.501032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:46607 (local address 127.24.167.62:34517)
0808 12:57:23.501268 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:23.501276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:23.502680 (+ 1404us) server_negotiation.cc:407] Connection header received
0808 12:57:23.504824 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:23.504828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:23.504908 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:23.505000 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:23.506458 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:23.507174 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:23.508726 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:23.509125 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:23.509235 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:23.512784 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:23.512789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:23.512792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:23.512825 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:23.516225 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:23.516911 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:23.516914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:23.516916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:23.517042 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:23.517524 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:23.517529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:23.517531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:23.517835 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:23.517966 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:23.518781 (+ 815us) server_negotiation.cc:299] Negotiation successful
0808 12:57:23.518965 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 12:57:23.520224 2394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:23.501551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.3:46607)
0808 12:57:23.502513 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:23.502558 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:23.504531 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:23.505199 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:23.505207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:23.505709 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:23.506267 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:23.506283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:23.507302 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:23.507306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:23.508571 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:23.508584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:23.509373 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:23.509379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:23.509577 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:23.510331 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:23.510365 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:23.512582 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:23.516369 (+ 3787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:23.516376 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:23.516389 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:23.516812 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:23.517180 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:23.517184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:23.517186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:23.517388 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:23.517972 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:23.517978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:23.518332 (+ 354us) client_negotiation.cc:769] Sending connection context
0808 12:57:23.518675 (+ 343us) client_negotiation.cc:240] Negotiation successful
0808 12:57:23.519090 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":376,"threads_started":1}
I20260808 12:57:23.521746 2391 heartbeater.cc:347] Connected to a master server at 127.24.167.62:34517
I20260808 12:57:23.522099 2391 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:23.522976 2391 heartbeater.cc:511] Master 127.24.167.62:34517 requested a full tablet report, sending...
I20260808 12:57:23.524408 1716 ts_manager.cc:194] Registered new tserver with Master: 62ff2f69928b42d3b99721fa260c0c5d (127.24.167.3:39209)
I20260808 12:57:23.525368 1716 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:46607
I20260808 12:57:23.528556 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.167.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.167.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.167.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.61.keytab.
Entry for principal oryx/127.24.167.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.61.keytab.
I20260808 12:57:23.581836 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.61:37779
--webserver_interface=127.24.167.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.61.keytab
--principal=oryx/127.24.167.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:34517,127.24.167.61:37779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:23.927520 2398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:23.927836 2398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:23.927991 2398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:23.935333 2398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:57:23.935437 2398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:23.935544 2398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:23.935644 2398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:57:23.935725 2398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:57:23.945861 2398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:34517,127.24.167.61:37779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.61.keytab
--principal=oryx/127.24.167.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.61:37779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.167.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
W20260808 12:57:23.946229 2398 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.24.167.62:34517,127.24.167.61:37779'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 12:57:23.947353 2398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:23.948657 2398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:23.960848 2411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:23.960918 2415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:23.960881 2410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:23.961879 2398 server_base.cc:1034] running on GCE node
I20260808 12:57:23.962710 2398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:23.964071 2398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:23.965308 2398 hybrid_clock.cc:648] HybridClock initialized: now 1786193843965218 us; error 57 us; skew 500 ppm
I20260808 12:57:23.965652 2398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:23 dist-test-slave-tk9z krb5kdc[1644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193843, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:23.968971 2398 init.cc:377] Logged in from keytab as oryx/127.24.167.61@KRBTEST.COM (short username oryx)
I20260808 12:57:23.970679 2398 webserver.cc:460] Webserver started at http://127.24.167.61:34279/ using document root <none> and password file <none>
I20260808 12:57:23.971477 2398 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:23.971588 2398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:23.971875 2398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:23.975081 2398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/data/instance:
uuid: "09d348e4a51a4834b6289b59a3af8e01"
format_stamp: "Formatted at 2026-08-08 12:57:23 on dist-test-slave-tk9z"
server_key: "00112d8a0d5d939b78ea4c9527a96d27"
server_key_iv: "99c0aadeeb01afca9e1c915965a73d21"
server_key_version: "encryptionkey@0"
I20260808 12:57:23.975900 2398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/wal/instance:
uuid: "09d348e4a51a4834b6289b59a3af8e01"
format_stamp: "Formatted at 2026-08-08 12:57:23 on dist-test-slave-tk9z"
server_key: "00112d8a0d5d939b78ea4c9527a96d27"
server_key_iv: "99c0aadeeb01afca9e1c915965a73d21"
server_key_version: "encryptionkey@0"
I20260808 12:57:23.981418 2398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 12:57:23.985149 2424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:23.986619 2398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:57:23.986809 2398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/wal
uuid: "09d348e4a51a4834b6289b59a3af8e01"
format_stamp: "Formatted at 2026-08-08 12:57:23 on dist-test-slave-tk9z"
server_key: "00112d8a0d5d939b78ea4c9527a96d27"
server_key_iv: "99c0aadeeb01afca9e1c915965a73d21"
server_key_version: "encryptionkey@0"
I20260808 12:57:23.987046 2398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:24.008123 2398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:24.012626 2190 heartbeater.cc:503] Master 127.24.167.62:34517 was elected leader, sending a full tablet report...
I20260808 12:57:24.012825 2398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:24.013100 2398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:24.032896 2398 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.61:37779
I20260808 12:57:24.032903 2500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.61:37779 every 8 connection(s)
I20260808 12:57:24.034282 2398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/data/info.pb
I20260808 12:57:24.034885 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 2398
I20260808 12:57:24.038995 2501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:24.045786 2501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 12:57:24 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.167.61@KRBTEST.COM
Aug 08 12:57:24 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193843, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.61@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
I20260808 12:57:24.059345 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.037337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.24.167.61:37779)
0808 12:57:24.038156 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:24.038167 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:24.038209 (+ 42us) server_negotiation.cc:407] Connection header received
0808 12:57:24.040145 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:24.040166 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:24.040521 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:24.040889 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:24.042431 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.043776 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.045036 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.045699 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.045905 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.051270 (+ 5365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:24.051278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:24.051287 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:24.051327 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:24.054428 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.055163 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.055174 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.055184 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.055529 (+ 345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.055915 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.055920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.055923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.056631 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:24.057040 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:24.057466 (+ 426us) server_negotiation.cc:299] Negotiation successful
0808 12:57:24.057974 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"spinlock_wait_cycles":17664,"thread_start_us":349,"threads_started":1}
I20260808 12:57:24.064234 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.048369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36034 (local address 127.24.167.62:34517)
0808 12:57:24.048855 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:24.048859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:24.049551 (+ 692us) server_negotiation.cc:407] Connection header received
0808 12:57:24.049751 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:24.049755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:24.049821 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:24.049917 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:24.051504 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.052441 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.053908 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.054315 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.054519 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.058460 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:24.058467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:24.058473 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:24.058517 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:24.061655 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.062487 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.062491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.062494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.062615 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.062972 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.062975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.062977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.063277 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:24.063391 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:24.063802 (+ 411us) server_negotiation.cc:299] Negotiation successful
0808 12:57:24.063979 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":274,"threads_started":1}
I20260808 12:57:24.064256 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.048343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36034)
0808 12:57:24.049350 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:24.049399 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:24.049603 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:24.050027 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:24.050037 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:24.050966 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:24.051340 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.051351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.052625 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.052629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.053746 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.053768 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.054581 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.054588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.054765 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.055373 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:24.055402 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:24.058221 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:24.061790 (+ 3569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.061800 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.061805 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.062314 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.062723 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.062727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.062729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.062864 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.063404 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:24.063410 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:24.063562 (+ 152us) client_negotiation.cc:769] Sending connection context
0808 12:57:24.063716 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 12:57:24.063969 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":655,"threads_started":1}
I20260808 12:57:24.066020 2501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 12:57:24 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193843, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.61@KRBTEST.COM for oryx/127.24.167.61@KRBTEST.COM
I20260808 12:57:24.078133 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.066910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:37779 (local address 127.0.0.1:59052)
0808 12:57:24.067079 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:24.067100 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:24.067195 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:24.067460 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:24.067463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:24.067764 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:24.067899 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.067906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.068838 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.068842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.069675 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.069685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.070301 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.070306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.070515 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.071000 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:24.071016 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:24.073066 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:24.075805 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.075813 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.075817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.076175 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.076574 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.076581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.076584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.076710 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.077182 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:24.077187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:24.077329 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 12:57:24.077466 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 12:57:24.077759 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 12:57:24.078222 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.066802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.24.167.61:37779)
0808 12:57:24.066950 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:24.066956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:24.067167 (+ 211us) server_negotiation.cc:407] Connection header received
0808 12:57:24.067272 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:24.067276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:24.067329 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:24.067404 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:24.068013 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.068708 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.069789 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.070170 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.070258 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.073233 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:24.073238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:24.073241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:24.073267 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:24.075678 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.076292 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.076299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.076303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.076464 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.076815 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.076821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.076824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.077067 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:24.077217 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:24.077545 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 12:57:24.077909 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":152,"server-negotiator.queue_time_us":52}
I20260808 12:57:24.083071 2501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: Bootstrap starting.
I20260808 12:57:24.086117 2501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:24.087225 2501 log.cc:824] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:24.090241 2501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: No bootstrap required, opened a new log
I20260808 12:57:24.097396 2501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } }
I20260808 12:57:24.097730 2501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:24.097826 2501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d348e4a51a4834b6289b59a3af8e01, State: Initialized, Role: FOLLOWER
I20260808 12:57:24.098531 2501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } }
I20260808 12:57:24.099398 2508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } } }
I20260808 12:57:24.099757 2508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 12:57:24.100380 2501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:57:24.104766 2508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:57:24.105170 2508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } }
I20260808 12:57:24.107024 2508 leader_election.cc:288] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2cb52625c1c4008a860a8dc90be27e5 (127.24.167.62:34517)
I20260808 12:57:24.107841 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09d348e4a51a4834b6289b59a3af8e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb52625c1c4008a860a8dc90be27e5" is_pre_election: true
I20260808 12:57:24.109004 2425 leader_election.cc:302] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 09d348e4a51a4834b6289b59a3af8e01; no voters: e2cb52625c1c4008a860a8dc90be27e5
I20260808 12:57:24.110291 2508 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 12:57:24.112247 2521 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 12:57:24.112383 2521 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 12:57:24.113238 2398 master_runner.cc:428] Detected that this master 09d348e4a51a4834b6289b59a3af8e01 is joining an existing cluster
I20260808 12:57:24.113344 2398 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.167.61:37779
I20260808 12:57:24.144435 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.126645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36042 (local address 127.24.167.62:34517)
0808 12:57:24.126855 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:24.126861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:24.126882 (+ 21us) server_negotiation.cc:407] Connection header received
0808 12:57:24.127025 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:24.127031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:24.127147 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:24.127251 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:24.129047 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.130170 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.132737 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.133536 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.133667 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.136760 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:24.136769 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:24.136777 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:24.136825 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:24.140937 (+ 4112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.141812 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.141817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.141820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.141967 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.142636 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.142640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.142644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.143059 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:24.143200 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:24.143850 (+ 650us) server_negotiation.cc:299] Negotiation successful
0808 12:57:24.144085 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 12:57:24.144464 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.127329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59062 (local address 127.24.167.61:37779)
0808 12:57:24.127496 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:24.127500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:24.128330 (+ 830us) server_negotiation.cc:407] Connection header received
0808 12:57:24.128543 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:24.128550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:24.128622 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:24.128702 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:24.130060 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.131237 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.133212 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.133941 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.134090 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.136765 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:24.136772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:24.136776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:24.136809 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:24.140791 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.141795 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.141800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.141803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.141967 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.142616 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.142621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.142624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.143082 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:24.143198 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:24.143826 (+ 628us) server_negotiation.cc:299] Negotiation successful
0808 12:57:24.144078 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
I20260808 12:57:24.145594 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.125947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36042)
0808 12:57:24.126678 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:24.126728 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:24.126877 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:24.127563 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:24.127567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:24.128415 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:24.128867 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.128880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.130485 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.130493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.132562 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.132581 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.134002 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.134011 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.134231 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.135019 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:24.135049 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:24.136538 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:24.141093 (+ 4555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.141100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.141106 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.141663 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.142115 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.142120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.142124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.142495 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.143279 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:24.143283 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:24.143569 (+ 286us) client_negotiation.cc:769] Sending connection context
0808 12:57:24.144755 (+ 1186us) client_negotiation.cc:240] Negotiation successful
0808 12:57:24.145082 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":42,"spinlock_wait_cycles":8064,"thread_start_us":741,"threads_started":1}
I20260808 12:57:24.146276 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.127434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:37779 (local address 127.0.0.1:59062)
0808 12:57:24.128241 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:24.128263 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:24.128414 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:24.128914 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:24.128921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:24.129575 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:24.129872 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.129884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.131450 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.131456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.133045 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.133061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.134452 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.134462 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.134669 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.135252 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:24.135279 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:24.136563 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:24.140948 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.140967 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.140972 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.141663 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.142115 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.142120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.142124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.142495 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.143279 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:24.143283 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:24.143547 (+ 264us) client_negotiation.cc:769] Sending connection context
0808 12:57:24.145152 (+ 1605us) client_negotiation.cc:240] Negotiation successful
0808 12:57:24.145466 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":53,"spinlock_wait_cycles":33664,"thread_start_us":407,"threads_started":1}
WARNING: no policy specified for oryx/127.24.167.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.167.60@KRBTEST.COM" created.
Aug 08 12:57:24 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193841, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.62@KRBTEST.COM for oryx/127.24.167.61@KRBTEST.COM
I20260808 12:57:24.169133 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.152703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.24.167.61:37779)
0808 12:57:24.153000 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:24.153007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:24.153418 (+ 411us) server_negotiation.cc:407] Connection header received
0808 12:57:24.153775 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:24.153781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:24.153883 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:24.154009 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:24.155606 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.156363 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.157682 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.158107 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.158247 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.161656 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:24.161663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:24.161668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:24.161706 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:24.165506 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.166558 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.166567 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.166572 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.166826 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.167330 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.167336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.167339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.167718 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:24.167878 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:24.168507 (+ 629us) server_negotiation.cc:299] Negotiation successful
0808 12:57:24.168748 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260808 12:57:24.169305 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.152682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:37779 (local address 127.0.0.1:59068)
0808 12:57:24.153286 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:24.153332 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:24.153550 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:24.154075 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:24.154082 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:24.155161 (+ 1079us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:24.155454 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.155465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.156515 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.156521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.157527 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.157539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.158276 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.158282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.158539 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.159203 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:24.159226 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:24.161461 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:24.165667 (+ 4206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.165684 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.165695 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.166407 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.166968 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.166976 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.166982 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.167179 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.167879 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:24.167889 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:24.168129 (+ 240us) client_negotiation.cc:769] Sending connection context
0808 12:57:24.168354 (+ 225us) client_negotiation.cc:240] Negotiation successful
0808 12:57:24.168747 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":333,"threads_started":1}
I20260808 12:57:24.171729 1716 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.167.61:37779: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01"
member_type: NON_VOTER
last_known_addr {
host: "127.24.167.61"
port: 37779
}
attrs {
promote: true
}
}
dest_uuid: "e2cb52625c1c4008a860a8dc90be27e5"
cas_config_opid_index: -1
I20260808 12:57:24.173144 1716 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: NON_VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: true } }
I20260808 12:57:24.179963 2544 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 09d348e4a51a4834b6289b59a3af8e01 (127.24.167.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: NON_VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: true } } }
I20260808 12:57:24.181607 2465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.167.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.60.keytab.
Entry for principal oryx/127.24.167.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.60.keytab.
I20260808 12:57:24.185458 2465 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 FOLLOWER]: Refusing update from remote peer e2cb52625c1c4008a860a8dc90be27e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 12:57:24.185421 2544 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.167.61:37779
I20260808 12:57:24.186496 2398 master.cc:537] Master@127.24.167.61:37779 shutting down...
I20260808 12:57:24.186529 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.60:35829
--webserver_interface=127.24.167.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.60.keytab
--principal=oryx/127.24.167.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:34517,127.24.167.61:37779,127.24.167.60:35829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 12:57:24.192422 2541 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: NON_VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: true } } }
I20260808 12:57:24.192811 2541 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:24.195253 2544 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: NON_VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:24.201156 2398 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260808 12:57:24.201310 1697 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260808 12:57:24.201758 2398 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:24.202047 2398 tablet_replica.cc:330] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: stopping tablet replica
I20260808 12:57:24.215605 2398 master.cc:552] Master@127.24.167.61:37779 shutdown complete.
I20260808 12:57:24.215791 2398 master_runner.cc:305] Clearing existing system tablet
I20260808 12:57:24.216452 2398 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:24.216586 2398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:24.221683 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.221065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:37779 (local address 127.0.0.1:59070)
0808 12:57:24.221313 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:24.221474 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260808 12:57:24.221966 1697 consensus_peers.cc:487] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 -> Peer 09d348e4a51a4834b6289b59a3af8e01 (127.24.167.61:37779): Couldn't send request to peer 09d348e4a51a4834b6289b59a3af8e01. Status: Network error: Client connection negotiation failed: client connection to 127.24.167.61:37779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 12:57:24.223847 2398 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 12:57:24.231653 2552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:24.232049 2398 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:24.232506 2398 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.004s
I20260808 12:57:24.232607 2398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/wal
uuid: "09d348e4a51a4834b6289b59a3af8e01"
format_stamp: "Formatted at 2026-08-08 12:57:23 on dist-test-slave-tk9z"
server_key: "00112d8a0d5d939b78ea4c9527a96d27"
server_key_iv: "99c0aadeeb01afca9e1c915965a73d21"
server_key_version: "encryptionkey@0"
I20260808 12:57:24.238067 2398 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:24.245424 2398 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 12:57:24.245635 2398 log.cc:1196] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 12:57:24.246121 2398 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: Deleting consensus metadata
I20260808 12:57:24.246438 2398 master_runner.cc:315] Copying system tablet from 127.24.167.62:34517
I20260808 12:57:24.251888 2398 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: tablet copy: Beginning tablet copy session from remote peer at address 127.24.167.62:34517
I20260808 12:57:24.268874 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.252819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.24.167.62:34517)
0808 12:57:24.253121 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:24.253128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:24.253670 (+ 542us) server_negotiation.cc:407] Connection header received
0808 12:57:24.253891 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:24.253897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:24.254000 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:24.254119 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:24.255276 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.256467 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.258438 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.259114 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.259253 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.261681 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:24.261691 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:24.261698 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:24.261751 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:24.265755 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.266639 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.266645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.266648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.266812 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.267262 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.267267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.267270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.267643 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:24.267808 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:24.268262 (+ 454us) server_negotiation.cc:299] Negotiation successful
0808 12:57:24.268503 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260808 12:57:24.268981 2562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.252750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36046)
0808 12:57:24.253516 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:24.253541 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:24.253733 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:24.254167 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:24.254173 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:24.254773 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:24.255076 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.255089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.256644 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.256651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.258222 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.258239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.259293 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.259300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.259510 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.260280 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:24.260304 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:24.261423 (+ 1119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:24.265907 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.265913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.265917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.266498 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.266937 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.266942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.266946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.267133 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.267786 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:24.267792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:24.267968 (+ 176us) client_negotiation.cc:769] Sending connection context
0808 12:57:24.268139 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 12:57:24.268502 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":487,"threads_started":1}
I20260808 12:57:24.270161 1752 tablet_copy_service.cc:140] P e2cb52625c1c4008a860a8dc90be27e5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 09d348e4a51a4834b6289b59a3af8e01 ({username='oryx', principal='oryx/127.24.167.61@KRBTEST.COM'} at 127.0.0.1:36046)
I20260808 12:57:24.270499 1752 tablet_copy_service.cc:161] P e2cb52625c1c4008a860a8dc90be27e5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 09d348e4a51a4834b6289b59a3af8e01 at {username='oryx', principal='oryx/127.24.167.61@KRBTEST.COM'} at 127.0.0.1:36046: session id = 09d348e4a51a4834b6289b59a3af8e01-00000000000000000000000000000000
I20260808 12:57:24.275053 1752 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 12:57:24.277362 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 12:57:24.284452 2398 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: tablet copy: Starting download of 0 data blocks...
I20260808 12:57:24.284813 2398 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: tablet copy: Starting download of 1 WAL segments...
I20260808 12:57:24.287762 2398 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 12:57:24.292423 1752 tablet_copy_service.cc:342] P e2cb52625c1c4008a860a8dc90be27e5: Request end of tablet copy session 09d348e4a51a4834b6289b59a3af8e01-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.167.61@KRBTEST.COM'} at 127.0.0.1:36046
I20260808 12:57:24.292622 1752 tablet_copy_service.cc:433] P e2cb52625c1c4008a860a8dc90be27e5: ending tablet copy session 09d348e4a51a4834b6289b59a3af8e01-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 09d348e4a51a4834b6289b59a3af8e01
I20260808 12:57:24.305075 2398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:24.307933 2564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:24.313994 2566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:24.321091 2570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:24.321959 2398 server_base.cc:1034] running on GCE node
I20260808 12:57:24.322431 2398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:24.323195 2398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:24.324506 2398 hybrid_clock.cc:648] HybridClock initialized: now 1786193844324413 us; error 36 us; skew 500 ppm
I20260808 12:57:24.324826 2398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:24 dist-test-slave-tk9z krb5kdc[1644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193844, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:24.327286 2398 init.cc:377] Logged in from keytab as oryx/127.24.167.61@KRBTEST.COM (short username oryx)
I20260808 12:57:24.328537 2398 webserver.cc:460] Webserver started at http://127.24.167.61:35297/ using document root <none> and password file <none>
I20260808 12:57:24.328912 2398 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:24.328999 2398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:24.332254 2398 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 12:57:24.334321 2579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:24.334965 2398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 12:57:24.335145 2398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/wal
uuid: "09d348e4a51a4834b6289b59a3af8e01"
format_stamp: "Formatted at 2026-08-08 12:57:23 on dist-test-slave-tk9z"
server_key: "00112d8a0d5d939b78ea4c9527a96d27"
server_key_iv: "99c0aadeeb01afca9e1c915965a73d21"
server_key_version: "encryptionkey@0"
I20260808 12:57:24.335319 2398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:24.392820 2398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:24.397557 2398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:24.422979 2398 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.61:37779
I20260808 12:57:24.423017 2653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.61:37779 every 8 connection(s)
I20260808 12:57:24.424384 2398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-1/data/info.pb
I20260808 12:57:24.430115 2654 sys_catalog.cc:263] Verifying existing consensus state
I20260808 12:57:24.434960 2654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: Bootstrap starting.
I20260808 12:57:24.454061 2654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 12:57:24.454855 2654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: Bootstrap complete.
I20260808 12:57:24.456339 2654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: NON_VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: true } }
I20260808 12:57:24.456790 2654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09d348e4a51a4834b6289b59a3af8e01, State: Initialized, Role: LEARNER
I20260808 12:57:24.457309 2654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: NON_VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: true } }
I20260808 12:57:24.458689 2659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: NON_VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: true } } }
I20260808 12:57:24.459589 2659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: This master's current role is: LEARNER
I20260808 12:57:24.459697 2654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:57:24.477981 2677 catalog_manager.cc:1259] Loaded cluster ID: ffe882dca60648c4b97a045851f266de
I20260808 12:57:24.478109 2677 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: loading cluster ID for follower catalog manager: success
I20260808 12:57:24.483573 2677 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: acquiring CA information for follower catalog manager: success
I20260808 12:57:24.487465 2677 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 12:57:24.528581 2391 heartbeater.cc:503] Master 127.24.167.62:34517 was elected leader, sending a full tablet report...
I20260808 12:57:24.659792 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.649999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59074 (local address 127.24.167.61:37779)
0808 12:57:24.651303 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:24.651308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:24.651345 (+ 37us) server_negotiation.cc:407] Connection header received
0808 12:57:24.651408 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:24.651411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:24.651477 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:24.651576 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:24.652206 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.653561 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.656921 (+ 3360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.658143 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.658244 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.658605 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:24.659179 (+ 574us) server_negotiation.cc:299] Negotiation successful
0808 12:57:24.659303 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1191,"thread_start_us":982,"threads_started":1}
I20260808 12:57:24.659842 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.649793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:37779 (local address 127.0.0.1:59074)
0808 12:57:24.650163 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:24.650197 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:24.650399 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:24.651692 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:24.651698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:24.651735 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:24.651959 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.651975 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.653717 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.653724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.656728 (+ 3004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.656758 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.658488 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.658498 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.658947 (+ 449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.659031 (+ 84us) client_negotiation.cc:769] Sending connection context
0808 12:57:24.659160 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 12:57:24.659398 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260808 12:57:24.661454 2620 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260808 12:57:24.662321 2659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2cb52625c1c4008a860a8dc90be27e5. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: NON_VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: true } } }
I20260808 12:57:24.662740 2659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: This master's current role is: LEARNER
W20260808 12:57:24.735669 2548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:24.736168 2548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:24.736445 2548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:24.746354 2548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:57:24.746559 2548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:24.746666 2548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:24.746774 2548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:57:24.746873 2548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:57:24.758625 2548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:34517,127.24.167.61:37779,127.24.167.60:35829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.60.keytab
--principal=oryx/127.24.167.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.60:35829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.167.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:24.760104 2548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:24.761782 2548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:24.779327 2685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:24.780277 2691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:24.780347 2686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:24.781994 2548 server_base.cc:1034] running on GCE node
I20260808 12:57:24.782943 2548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:24.784672 2548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:24.785955 2548 hybrid_clock.cc:648] HybridClock initialized: now 1786193844785803 us; error 93 us; skew 500 ppm
I20260808 12:57:24.786458 2548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:24 dist-test-slave-tk9z krb5kdc[1644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193844, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:24.790896 2548 init.cc:377] Logged in from keytab as oryx/127.24.167.60@KRBTEST.COM (short username oryx)
I20260808 12:57:24.793012 2548 webserver.cc:460] Webserver started at http://127.24.167.60:37243/ using document root <none> and password file <none>
I20260808 12:57:24.793907 2548 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:24.794057 2548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:24.794474 2548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:24.798024 2548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/data/instance:
uuid: "e2bfd2b497ef40f9a6a450d9f11467fc"
format_stamp: "Formatted at 2026-08-08 12:57:24 on dist-test-slave-tk9z"
server_key: "b8ee302ba0287ae5a48abc8aff09be51"
server_key_iv: "dea333deb2768cb6003b0373302a116c"
server_key_version: "encryptionkey@0"
I20260808 12:57:24.799005 2548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/wal/instance:
uuid: "e2bfd2b497ef40f9a6a450d9f11467fc"
format_stamp: "Formatted at 2026-08-08 12:57:24 on dist-test-slave-tk9z"
server_key: "b8ee302ba0287ae5a48abc8aff09be51"
server_key_iv: "dea333deb2768cb6003b0373302a116c"
server_key_version: "encryptionkey@0"
I20260808 12:57:24.805316 2548 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260808 12:57:24.809473 2701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:24.811088 2548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:57:24.811318 2548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/wal
uuid: "e2bfd2b497ef40f9a6a450d9f11467fc"
format_stamp: "Formatted at 2026-08-08 12:57:24 on dist-test-slave-tk9z"
server_key: "b8ee302ba0287ae5a48abc8aff09be51"
server_key_iv: "dea333deb2768cb6003b0373302a116c"
server_key_version: "encryptionkey@0"
I20260808 12:57:24.811568 2548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:24.834117 2548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:24.838909 2548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:24.839305 2548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:24.878304 2548 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.60:35829
I20260808 12:57:24.878468 2777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.60:35829 every 8 connection(s)
I20260808 12:57:24.879999 2548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/data/info.pb
I20260808 12:57:24.884416 2778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:24.887120 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 2548
I20260808 12:57:24.893689 2778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 12:57:24 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.167.60@KRBTEST.COM
Aug 08 12:57:24 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193844, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.60@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
I20260808 12:57:24.923770 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.890112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36940 (local address 127.24.167.60:35829)
0808 12:57:24.891600 (+ 1488us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:24.891613 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:24.891657 (+ 44us) server_negotiation.cc:407] Connection header received
0808 12:57:24.894526 (+ 2869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:24.894551 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:24.895058 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:24.895462 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:24.897334 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.900426 (+ 3092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.902115 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.903050 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.903188 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.911441 (+ 8253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:24.911458 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:24.911480 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:24.911558 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:24.917348 (+ 5790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.918728 (+ 1380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.918738 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.918750 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.919084 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.919662 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.919668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.919671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.920699 (+ 1028us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:24.921159 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:24.921542 (+ 383us) server_negotiation.cc:299] Negotiation successful
0808 12:57:24.922152 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1238,"thread_start_us":846,"threads_started":1}
I20260808 12:57:24.925689 2783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.897763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.24.167.62:34517)
0808 12:57:24.898803 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:24.898823 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:24.899736 (+ 913us) server_negotiation.cc:407] Connection header received
0808 12:57:24.899959 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:24.899964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:24.900042 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:24.900169 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:24.902612 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.903988 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.905871 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.906812 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.906972 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.913631 (+ 6659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:24.913660 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:24.913668 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:24.913745 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:24.921947 (+ 8202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.922811 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.922818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.922822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.923032 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.923495 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.923500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.923503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.924085 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:24.924258 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:24.924840 (+ 582us) server_negotiation.cc:299] Negotiation successful
0808 12:57:24.925091 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":861,"thread_start_us":706,"threads_started":1}
I20260808 12:57:24.925864 2784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.897878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36060)
0808 12:57:24.899571 (+ 1693us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:24.899627 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:24.899806 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:24.900326 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:24.900335 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:24.901685 (+ 1350us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:24.902263 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.902277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.904188 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.904198 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.905680 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.905694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.907382 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.907392 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.907727 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.908785 (+ 1058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:24.908837 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:24.913366 (+ 4529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:24.922089 (+ 8723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.922099 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.922105 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.922656 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.923163 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.923166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.923169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.923355 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.924223 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:24.924229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:24.924395 (+ 166us) client_negotiation.cc:769] Sending connection context
0808 12:57:24.924566 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 12:57:24.925082 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1395,"thread_start_us":1056,"threads_started":1}
I20260808 12:57:24.927363 2778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 12:57:24 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193844, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.60@KRBTEST.COM for oryx/127.24.167.61@KRBTEST.COM
I20260808 12:57:24.944109 2784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.928164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:37779 (local address 127.0.0.1:59082)
0808 12:57:24.928417 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:24.928434 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:24.928520 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:24.929025 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:24.929030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:24.929556 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:24.929807 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.929817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.931145 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.931154 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.932492 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.932506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.933431 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.933442 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.933844 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.934568 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:24.934585 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:24.937043 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:24.940887 (+ 3844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.940895 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.940900 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.941440 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.941911 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.941918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.941922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.942132 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.942790 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:24.942796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:24.942990 (+ 194us) client_negotiation.cc:769] Sending connection context
0808 12:57:24.943137 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 12:57:24.943546 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 12:57:24.944132 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.928525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.24.167.61:37779)
0808 12:57:24.928738 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:24.928743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:24.928758 (+ 15us) server_negotiation.cc:407] Connection header received
0808 12:57:24.928815 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:24.928819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:24.928880 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:24.928999 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:24.929966 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.930998 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.932653 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.933184 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.933356 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.937276 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:24.937282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:24.937286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:24.937318 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:24.940750 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.941565 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.941572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.941575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.941788 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.942252 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.942258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.942261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.942663 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:24.942871 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:24.943232 (+ 361us) server_negotiation.cc:299] Negotiation successful
0808 12:57:24.943546 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 12:57:24.945334 2778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 12:57:24 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193844, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.60@KRBTEST.COM for oryx/127.24.167.60@KRBTEST.COM
I20260808 12:57:24.960728 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.946116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.24.167.60:35829)
0808 12:57:24.946323 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:24.946329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:24.946525 (+ 196us) server_negotiation.cc:407] Connection header received
0808 12:57:24.946709 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:24.946712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:24.946761 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:24.946854 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:24.947765 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.948513 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.949853 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.950527 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.950681 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.953947 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:24.953952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:24.953955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:24.953981 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:24.957174 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.958067 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.958076 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.958080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.958251 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:24.958757 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:24.958775 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:24.958782 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:24.959150 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:24.959315 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:24.959837 (+ 522us) server_negotiation.cc:299] Negotiation successful
0808 12:57:24.960214 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":75}
I20260808 12:57:24.960794 2784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:24.946140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35829 (local address 127.0.0.1:36942)
0808 12:57:24.946441 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:24.946465 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:24.946585 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:24.946873 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:24.946877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:24.947407 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:24.947629 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.947638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.948641 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.948648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.949709 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:24.949722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:24.950687 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:24.950692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:24.951078 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:24.951631 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:24.951648 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:24.953712 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:24.957334 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.957341 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.957345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.957928 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.958394 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:24.958402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:24.958406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:24.958641 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:24.959317 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:24.959322 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:24.959523 (+ 201us) client_negotiation.cc:769] Sending connection context
0808 12:57:24.959681 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 12:57:24.960001 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":3}
I20260808 12:57:24.966876 2778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: Bootstrap starting.
I20260808 12:57:24.970940 2778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:24.972597 2778 log.cc:824] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:24.976225 2778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: No bootstrap required, opened a new log
I20260808 12:57:24.986533 2778 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } }
I20260808 12:57:24.987049 2778 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:24.987288 2778 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2bfd2b497ef40f9a6a450d9f11467fc, State: Initialized, Role: FOLLOWER
I20260808 12:57:24.987978 2778 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } }
I20260808 12:57:24.989401 2793 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } } }
I20260808 12:57:24.990000 2793 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [sys.catalog]: This master's current role is: FOLLOWER
I20260808 12:57:24.990563 2778 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [sys.catalog]: configured and running, proceeding with master startup.
W20260808 12:57:25.004119 2809 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 12:57:25.004380 2809 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 12:57:25.007252 2793 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:57:25.007932 2793 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } }
I20260808 12:57:25.010211 2793 leader_election.cc:288] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2cb52625c1c4008a860a8dc90be27e5 (127.24.167.62:34517), 09d348e4a51a4834b6289b59a3af8e01 (127.24.167.61:37779)
I20260808 12:57:25.010157 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb52625c1c4008a860a8dc90be27e5" is_pre_election: true
I20260808 12:57:25.010452 2620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09d348e4a51a4834b6289b59a3af8e01" is_pre_election: true
I20260808 12:57:25.011330 2548 master_runner.cc:428] Detected that this master e2bfd2b497ef40f9a6a450d9f11467fc is joining an existing cluster
I20260808 12:57:25.011482 2548 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.167.60:35829
I20260808 12:57:25.011400 2706 leader_election.cc:302] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2bfd2b497ef40f9a6a450d9f11467fc; no voters: 09d348e4a51a4834b6289b59a3af8e01, e2cb52625c1c4008a860a8dc90be27e5
I20260808 12:57:25.012022 2793 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 12:57:25.031667 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 1815
Aug 08 12:57:25 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.167.1@KRBTEST.COM
I20260808 12:57:25.069427 2783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.022013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36072 (local address 127.24.167.62:34517)
0808 12:57:25.022209 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:25.022215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:25.030160 (+ 7945us) server_negotiation.cc:407] Connection header received
0808 12:57:25.031875 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:25.031881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:25.032026 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:25.032134 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:25.037698 (+ 5564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.039029 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.045451 (+ 6422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.046781 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.046898 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.059662 (+ 12764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:25.059673 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:25.059681 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:25.059757 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:25.064394 (+ 4637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.065852 (+ 1458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.065858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.065861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.066067 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.066715 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.066720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.066723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.067206 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:25.067349 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:25.068627 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0808 12:57:25.068924 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 12:57:25.070744 2828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.033024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36766 (local address 127.24.167.1:32969)
0808 12:57:25.034241 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:25.034255 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:25.034294 (+ 39us) server_negotiation.cc:407] Connection header received
0808 12:57:25.034521 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:25.034535 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:25.034949 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:25.037910 (+ 2961us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:25.038347 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.040758 (+ 2411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.043609 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.045084 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.046130 (+ 1046us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.055203 (+ 9073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:25.055222 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:25.055243 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:25.055285 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:25.060627 (+ 5342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.065906 (+ 5279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.065919 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.065924 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.066146 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.066867 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.066872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.066875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.067498 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:25.067951 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:25.069808 (+ 1857us) server_negotiation.cc:299] Negotiation successful
0808 12:57:25.070204 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":682,"threads_started":1}
I20260808 12:57:25.071081 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.021303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36072)
0808 12:57:25.030051 (+ 8748us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:25.030073 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:25.031446 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:25.032430 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:25.032441 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:25.034808 (+ 2367us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:25.037076 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.037097 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.039400 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.039406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.041857 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.041872 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.050916 (+ 9044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.050928 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.051257 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.054696 (+ 3439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:25.054729 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:25.056420 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:25.065154 (+ 8734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.065160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.065164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.065732 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.066240 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.066244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.066247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.066546 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.067463 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:25.067468 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:25.067882 (+ 414us) client_negotiation.cc:769] Sending connection context
0808 12:57:25.068046 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 12:57:25.068418 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8495,"thread_start_us":12060,"threads_started":1}
I20260808 12:57:25.073040 1992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:25.082418 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.021343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.24.167.61:37779)
0808 12:57:25.021867 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:25.021876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:25.044931 (+ 23055us) server_negotiation.cc:407] Connection header received
0808 12:57:25.052010 (+ 7079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:25.052021 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:25.052146 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:25.052237 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:25.057582 (+ 5345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.059199 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.063345 (+ 4146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.064308 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.064421 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.071965 (+ 7544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:25.071972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:25.071977 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:25.072023 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:25.077316 (+ 5293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.078573 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.078582 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.078588 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.078861 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.079566 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.079574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.079580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.080133 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:25.080306 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:25.081249 (+ 943us) server_negotiation.cc:299] Negotiation successful
0808 12:57:25.081701 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400}
I20260808 12:57:25.086884 2544 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: NON_VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: true } } }
I20260808 12:57:25.087304 2544 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:25.087926 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.022282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:37779 (local address 127.0.0.1:59090)
0808 12:57:25.044811 (+ 22529us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:25.044843 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:25.051785 (+ 6942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:25.052597 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:25.052604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:25.054184 (+ 1580us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:25.057060 (+ 2876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.057078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.059417 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.059421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.063159 (+ 3738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.063177 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.064827 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.064834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.065203 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.069978 (+ 4775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:25.070002 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:25.071693 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:25.077520 (+ 5827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.077529 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.077536 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.078350 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.079048 (+ 698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.079056 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.079062 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.079402 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.080409 (+ 1007us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:25.080416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:25.080813 (+ 397us) client_negotiation.cc:769] Sending connection context
0808 12:57:25.082680 (+ 1867us) client_negotiation.cc:240] Negotiation successful
0808 12:57:25.083184 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22344,"spinlock_wait_cycles":71040,"thread_start_us":22659,"threads_started":1}
I20260808 12:57:25.090873 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.022532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35829 (local address 127.0.0.1:36946)
0808 12:57:25.059539 (+ 37007us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:25.059559 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:25.060471 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:25.061059 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:25.061065 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:25.062228 (+ 1163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:25.062978 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.062993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.065144 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.065150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.067725 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.067746 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.069628 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.069635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.069832 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.071450 (+ 1618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:25.071474 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:25.075200 (+ 3726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:25.086629 (+ 11429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.086636 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.086639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.087394 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.088221 (+ 827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.088225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.088229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.088445 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.089359 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:25.089363 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:25.089632 (+ 269us) client_negotiation.cc:769] Sending connection context
0808 12:57:25.089817 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 12:57:25.090156 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36793,"thread_start_us":36308,"threads_started":1}
W20260808 12:57:25.091843 2709 tablet.cc:2369] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:57:25.093087 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.021334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36946 (local address 127.24.167.60:35829)
0808 12:57:25.021702 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:25.021710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:25.059641 (+ 37931us) server_negotiation.cc:407] Connection header received
0808 12:57:25.060663 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:25.060670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:25.060749 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:25.060811 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:25.063148 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.064714 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.067892 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.069079 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.069205 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.075395 (+ 6190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:25.075401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:25.075405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:25.075439 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:25.086494 (+ 11055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.087578 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.087586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.087592 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.087932 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.088565 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.088570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.088574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.089086 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:25.089301 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:25.091256 (+ 1955us) server_negotiation.cc:299] Negotiation successful
0808 12:57:25.091588 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
Aug 08 12:57:25 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193841, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.62@KRBTEST.COM for oryx/127.24.167.60@KRBTEST.COM
I20260808 12:57:25.107331 2832 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5: attempting to promote NON_VOTER 09d348e4a51a4834b6289b59a3af8e01 to VOTER
I20260808 12:57:25.109129 2832 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } }
I20260808 12:57:25.112699 2620 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 LEARNER]: Refusing update from remote peer e2cb52625c1c4008a860a8dc90be27e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 12:57:25.113517 2544 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 12:57:25.118983 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.082123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35829 (local address 127.0.0.1:36954)
0808 12:57:25.082686 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:25.082710 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:25.082835 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:25.095418 (+ 12583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:25.095424 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:25.096239 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:25.096597 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.096616 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.099723 (+ 3107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.099728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.101513 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.101530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.103160 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.103168 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.103393 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.104266 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:25.104303 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:25.108481 (+ 4178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:25.115833 (+ 7352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.115840 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.115844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.116451 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.116923 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.116929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.116933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.117126 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.117856 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:25.117862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:25.118087 (+ 225us) client_negotiation.cc:769] Sending connection context
0808 12:57:25.118233 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 12:57:25.118593 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392}
I20260808 12:57:25.120254 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.082608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36954 (local address 127.24.167.60:35829)
0808 12:57:25.094988 (+ 12380us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:25.094996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:25.095020 (+ 24us) server_negotiation.cc:407] Connection header received
0808 12:57:25.095130 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:25.095138 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:25.095241 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:25.095357 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:25.096825 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.099528 (+ 2703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.101781 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.102863 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.106345 (+ 3482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.108811 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:25.108819 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:25.108824 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:25.108866 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:25.115655 (+ 6789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.116631 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.116638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.116642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.116787 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.117280 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.117297 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.117300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.117699 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:25.119384 (+ 1685us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:25.119509 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 12:57:25.119778 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12247,"thread_start_us":15701,"threads_started":1}
I20260808 12:57:25.122318 1716 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.167.60:35829: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc"
member_type: NON_VOTER
last_known_addr {
host: "127.24.167.60"
port: 35829
}
attrs {
promote: true
}
}
dest_uuid: "e2cb52625c1c4008a860a8dc90be27e5"
cas_config_opid_index: 5
W20260808 12:57:25.123153 1716 master_service.cc:301] AddMaster 127.24.167.60:35829 failed with retriable error: Illegal state: Failed initiating master Raft ChangeConfig request, error: unknown: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: NON_VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } }
W20260808 12:57:25.126482 2812 master_proxy_rpc.cc:203] Re-attempting AddMaster request to leader Master (127.24.167.62:34517)
I20260808 12:57:25.131760 2832 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 09d348e4a51a4834b6289b59a3af8e01 (127.24.167.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } }
I20260808 12:57:25.137777 1716 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.167.60:35829: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc"
member_type: NON_VOTER
last_known_addr {
host: "127.24.167.60"
port: 35829
}
attrs {
promote: true
}
}
dest_uuid: "e2cb52625c1c4008a860a8dc90be27e5"
cas_config_opid_index: 6
I20260808 12:57:25.137364 2834 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } }
I20260808 12:57:25.138217 2834 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:25.137926 2620 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 09d348e4a51a4834b6289b59a3af8e01 (127.24.167.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } }
I20260808 12:57:25.138877 1716 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: NON_VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: true } }
I20260808 12:57:25.142048 2744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:25.137086 2832 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } }
I20260808 12:57:25.145076 2832 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:25.145800 2659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } }
I20260808 12:57:25.146049 2659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 12:57:25.146677 2619 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 FOLLOWER]: Refusing update from remote peer e2cb52625c1c4008a860a8dc90be27e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 12:57:25.148082 2832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 12:57:25.151075 2744 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 1 FOLLOWER]: Refusing update from remote peer e2cb52625c1c4008a860a8dc90be27e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 12:57:25.151327 2834 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e2bfd2b497ef40f9a6a450d9f11467fc (127.24.167.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: NON_VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: true } } }
I20260808 12:57:25.152164 2619 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e2bfd2b497ef40f9a6a450d9f11467fc (127.24.167.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: NON_VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: true } } }
I20260808 12:57:25.156356 2834 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.167.60:35829
I20260808 12:57:25.156926 2548 master.cc:537] Master@127.24.167.60:35829 shutting down...
I20260808 12:57:25.156898 2544 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: NON_VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: true } } }
I20260808 12:57:25.157207 2544 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:25.159761 2659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: NON_VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: true } } }
I20260808 12:57:25.160121 2659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 12:57:25.160778 2832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: NON_VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:25.161401 2834 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: NON_VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: true } } }
I20260808 12:57:25.170310 2834 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: This master's current role is: LEADER
W20260808 12:57:25.171726 1695 consensus_peers.cc:487] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 -> Peer e2bfd2b497ef40f9a6a450d9f11467fc (127.24.167.60:35829): Couldn't send request to peer e2bfd2b497ef40f9a6a450d9f11467fc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 12:57:25.192868 2548 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:25.193455 2548 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:25.193732 2548 tablet_replica.cc:330] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: stopping tablet replica
I20260808 12:57:25.202209 2548 master.cc:552] Master@127.24.167.60:35829 shutdown complete.
I20260808 12:57:25.202307 2548 master_runner.cc:305] Clearing existing system tablet
I20260808 12:57:25.202701 2548 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:25.202838 2548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:25.205803 2548 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 12:57:25.207572 2845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:25.208307 2548 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:25.208675 2548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 12:57:25.208808 2548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/wal
uuid: "e2bfd2b497ef40f9a6a450d9f11467fc"
format_stamp: "Formatted at 2026-08-08 12:57:24 on dist-test-slave-tk9z"
server_key: "b8ee302ba0287ae5a48abc8aff09be51"
server_key_iv: "dea333deb2768cb6003b0373302a116c"
server_key_version: "encryptionkey@0"
I20260808 12:57:25.214654 2548 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:25.221690 2548 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 12:57:25.221858 2548 log.cc:1196] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 12:57:25.222347 2548 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: Deleting consensus metadata
I20260808 12:57:25.222625 2548 master_runner.cc:315] Copying system tablet from 127.24.167.62:34517
I20260808 12:57:25.227197 2548 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: tablet copy: Beginning tablet copy session from remote peer at address 127.24.167.62:34517
I20260808 12:57:25.244002 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 1815
I20260808 12:57:25.249641 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.228192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36086)
0808 12:57:25.229359 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:25.229408 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:25.229605 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:25.230115 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:25.230120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:25.230978 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:25.231533 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.231551 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.233433 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.233439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.235580 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.235598 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.236912 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.236918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.237149 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.238041 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:25.238064 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:25.240978 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:25.246062 (+ 5084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.246068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.246073 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.246714 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.247286 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.247291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.247296 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.247553 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.248320 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:25.248325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:25.248577 (+ 252us) client_negotiation.cc:769] Sending connection context
0808 12:57:25.248727 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 12:57:25.249105 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"thread_start_us":680,"threads_started":1}
I20260808 12:57:25.250928 2783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.228255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36086 (local address 127.24.167.62:34517)
0808 12:57:25.228446 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:25.228451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:25.229547 (+ 1096us) server_negotiation.cc:407] Connection header received
0808 12:57:25.229786 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:25.229794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:25.229935 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:25.230066 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:25.231721 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.233227 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.235737 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.236675 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.236867 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.241296 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:25.241306 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:25.241312 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:25.241380 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:25.245847 (+ 4467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.246909 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.246924 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.246928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.247120 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.247701 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.247706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.247712 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.248102 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:25.248254 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:25.250106 (+ 1852us) server_negotiation.cc:299] Negotiation successful
0808 12:57:25.250417 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 12:57:25.252717 1752 tablet_copy_service.cc:140] P e2cb52625c1c4008a860a8dc90be27e5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e2bfd2b497ef40f9a6a450d9f11467fc ({username='oryx', principal='oryx/127.24.167.60@KRBTEST.COM'} at 127.0.0.1:36086)
I20260808 12:57:25.252931 1752 tablet_copy_service.cc:161] P e2cb52625c1c4008a860a8dc90be27e5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e2bfd2b497ef40f9a6a450d9f11467fc at {username='oryx', principal='oryx/127.24.167.60@KRBTEST.COM'} at 127.0.0.1:36086: session id = e2bfd2b497ef40f9a6a450d9f11467fc-00000000000000000000000000000000
I20260808 12:57:25.257880 1752 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 12:57:25.260193 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 12:57:25.269164 2548 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: tablet copy: Starting download of 0 data blocks...
I20260808 12:57:25.269537 2548 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: tablet copy: Starting download of 1 WAL segments...
I20260808 12:57:25.272894 2548 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 12:57:25.277190 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:32969
--local_ip_for_outbound_sockets=127.24.167.1
--tserver_master_addrs=127.24.167.62:34517
--webserver_port=35375
--webserver_interface=127.24.167.1
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.1.keytab
--principal=oryx/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.167.62:34517,127.24.167.61:37779
--tserver_master_addrs=127.24.167.62:34517,127.24.167.61:37779,127.24.167.60:35829 with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 12:57:25.278013 1752 tablet_copy_service.cc:342] P e2cb52625c1c4008a860a8dc90be27e5: Request end of tablet copy session e2bfd2b497ef40f9a6a450d9f11467fc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.167.60@KRBTEST.COM'} at 127.0.0.1:36086
I20260808 12:57:25.278167 1752 tablet_copy_service.cc:433] P e2cb52625c1c4008a860a8dc90be27e5: ending tablet copy session e2bfd2b497ef40f9a6a450d9f11467fc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e2bfd2b497ef40f9a6a450d9f11467fc
I20260808 12:57:25.290843 2548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:25.293545 2857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:25.294328 2859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:25.302044 2861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:25.302593 2548 server_base.cc:1034] running on GCE node
I20260808 12:57:25.302971 2548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:25.303697 2548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:25.304978 2548 hybrid_clock.cc:648] HybridClock initialized: now 1786193845304940 us; error 31 us; skew 500 ppm
I20260808 12:57:25.305228 2548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:25 dist-test-slave-tk9z krb5kdc[1644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193845, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:25.307216 2548 init.cc:377] Logged in from keytab as oryx/127.24.167.60@KRBTEST.COM (short username oryx)
I20260808 12:57:25.315054 2548 webserver.cc:460] Webserver started at http://127.24.167.60:43445/ using document root <none> and password file <none>
I20260808 12:57:25.315378 2548 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:25.315493 2548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:25.317924 2548 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 12:57:25.319679 2871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:25.320145 2548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 12:57:25.320276 2548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/wal
uuid: "e2bfd2b497ef40f9a6a450d9f11467fc"
format_stamp: "Formatted at 2026-08-08 12:57:24 on dist-test-slave-tk9z"
server_key: "b8ee302ba0287ae5a48abc8aff09be51"
server_key_iv: "dea333deb2768cb6003b0373302a116c"
server_key_version: "encryptionkey@0"
I20260808 12:57:25.320477 2548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:25.340076 2548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:25.343700 2548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:25.403191 2548 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.60:35829
I20260808 12:57:25.403290 2955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.60:35829 every 8 connection(s)
I20260808 12:57:25.404255 2548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/master-2/data/info.pb
I20260808 12:57:25.409852 2957 sys_catalog.cc:263] Verifying existing consensus state
I20260808 12:57:25.414417 2957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: Bootstrap starting.
I20260808 12:57:25.435877 2957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 12:57:25.436592 2957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: Bootstrap complete.
I20260808 12:57:25.437762 2957 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: NON_VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: true } }
I20260808 12:57:25.438258 2957 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e2bfd2b497ef40f9a6a450d9f11467fc, State: Initialized, Role: LEARNER
I20260808 12:57:25.438589 2957 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: NON_VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: true } }
I20260808 12:57:25.445020 2963 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: NON_VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: true } } }
I20260808 12:57:25.445392 2957 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:57:25.445470 2963 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [sys.catalog]: This master's current role is: LEARNER
I20260808 12:57:25.456107 2986 catalog_manager.cc:1259] Loaded cluster ID: ffe882dca60648c4b97a045851f266de
I20260808 12:57:25.456220 2986 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: loading cluster ID for follower catalog manager: success
I20260808 12:57:25.460311 2986 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: acquiring CA information for follower catalog manager: success
I20260808 12:57:25.462860 2986 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 12:57:25.696866 2989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.689039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36960 (local address 127.24.167.60:35829)
0808 12:57:25.689560 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:25.689564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:25.689583 (+ 19us) server_negotiation.cc:407] Connection header received
0808 12:57:25.689812 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:25.689816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:25.689888 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:25.689996 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:25.690737 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.692010 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.694591 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.695644 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.695772 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.696019 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:25.696358 (+ 339us) server_negotiation.cc:299] Negotiation successful
0808 12:57:25.696505 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":291,"threads_started":1}
I20260808 12:57:25.697089 2988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.688720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35829 (local address 127.0.0.1:36960)
0808 12:57:25.689338 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:25.689379 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:25.689622 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:25.690094 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:25.690101 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:25.690158 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:25.690583 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.690598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.692225 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.692233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.694430 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.694447 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.695876 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.695885 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.696130 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.696192 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 12:57:25.696349 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 12:57:25.696567 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":279,"threads_started":1}
I20260808 12:57:25.699007 2916 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260808 12:57:25.699990 2963 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2cb52625c1c4008a860a8dc90be27e5. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: NON_VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: true } } }
I20260808 12:57:25.700397 2963 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [sys.catalog]: This master's current role is: LEARNER
W20260808 12:57:25.742965 2855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:25.743242 2855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:25.743466 2855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:25.752063 2855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:25.752243 2855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:25.752445 2855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:57:25.764173 2855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.1.keytab
--principal=oryx/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:32969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=35375
--tserver_master_addrs=127.24.167.62:34517,127.24.167.61:37779,127.24.167.60:35829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:25.765972 2855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:25.767673 2855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:25.783226 2994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:25.783541 2995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:25.784744 2998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:25.785416 2855 server_base.cc:1034] running on GCE node
I20260808 12:57:25.786131 2855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:25.787205 2855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:25.788420 2855 hybrid_clock.cc:648] HybridClock initialized: now 1786193845788360 us; error 32 us; skew 500 ppm
I20260808 12:57:25.788757 2855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:25 dist-test-slave-tk9z krb5kdc[1644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193845, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:25.792477 2855 init.cc:377] Logged in from keytab as oryx/127.24.167.1@KRBTEST.COM (short username oryx)
I20260808 12:57:25.794512 2855 webserver.cc:460] Webserver started at http://127.24.167.1:35375/ using document root <none> and password file <none>
I20260808 12:57:25.795274 2855 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:25.795399 2855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:25.801745 2855 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.005s
I20260808 12:57:25.804975 3007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:25.806545 2855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 12:57:25.806751 2855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "bc076c248e2541029f9460a5295055b9"
format_stamp: "Formatted at 2026-08-08 12:57:22 on dist-test-slave-tk9z"
server_key: "75adc5363817b8e437c567d9052fe088"
server_key_iv: "fa31ceb18fd623500f12bec6a2a96a55"
server_key_version: "encryptionkey@0"
I20260808 12:57:25.808128 2855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:25.823674 2855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:25.829663 2855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:25.830163 2855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:25.831935 2855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:25.833920 2855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:25.834129 2855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:25.834280 2855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:25.834417 2855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:25.907647 2855 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:32969
I20260808 12:57:25.907681 3196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:32969 every 8 connection(s)
I20260808 12:57:25.909703 2855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:57:25.912875 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 2855
I20260808 12:57:25.913384 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 2006
Aug 08 12:57:25 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.167.2@KRBTEST.COM
I20260808 12:57:25.954499 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.914811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.24.167.2:34171)
0808 12:57:25.916128 (+ 1317us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:25.916135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:25.920808 (+ 4673us) server_negotiation.cc:407] Connection header received
0808 12:57:25.921145 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:25.921155 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:25.921470 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:25.921646 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:25.925785 (+ 4139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.928343 (+ 2558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.931154 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.932346 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.932475 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.942546 (+ 10071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:25.942562 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:25.942575 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:25.942608 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:25.948576 (+ 5968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.950050 (+ 1474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.950072 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.950076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.950403 (+ 327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.951060 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.951065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.951068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.951625 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:25.952022 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:25.953394 (+ 1372us) server_negotiation.cc:299] Negotiation successful
0808 12:57:25.953715 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1117,"thread_start_us":703,"threads_started":1}
I20260808 12:57:25.956419 2180 generic_service.cc:196] Checking for leaks (request via RPC)
Aug 08 12:57:25 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193845, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
Aug 08 12:57:25 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193845, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.60@KRBTEST.COM
Aug 08 12:57:25 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193845, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.61@KRBTEST.COM
I20260808 12:57:25.973173 2989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.920114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:46193 (local address 127.24.167.60:35829)
0808 12:57:25.920397 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:25.920404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:25.939174 (+ 18770us) server_negotiation.cc:407] Connection header received
0808 12:57:25.943192 (+ 4018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:25.943199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:25.943323 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:25.943412 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:25.947569 (+ 4157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.949053 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.953431 (+ 4378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.954239 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.954425 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.963498 (+ 9073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:25.963505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:25.963509 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:25.963555 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:25.968310 (+ 4755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.969680 (+ 1370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.969685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.969689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.969912 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.970660 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.970668 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.970671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.971218 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:25.971364 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:25.972510 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0808 12:57:25.972742 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 12:57:25.973238 3208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.914785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:42189 (local address 127.24.167.61:37779)
0808 12:57:25.919257 (+ 4472us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:25.919264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:25.940027 (+ 20763us) server_negotiation.cc:407] Connection header received
0808 12:57:25.943157 (+ 3130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:25.943163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:25.943306 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:25.943389 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:25.947360 (+ 3971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.949080 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.953314 (+ 4234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.954082 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.954185 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.963910 (+ 9725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:25.963917 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:25.963921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:25.963965 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:25.968749 (+ 4784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.970055 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.970060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.970063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.970300 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.970888 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.970893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.970897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.971264 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:25.971374 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:25.972514 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0808 12:57:25.972742 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4309,"thread_start_us":4145,"threads_started":1}
I20260808 12:57:25.973513 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.914403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:47021 (local address 127.24.167.62:34517)
0808 12:57:25.919913 (+ 5510us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:25.919921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:25.939077 (+ 19156us) server_negotiation.cc:407] Connection header received
0808 12:57:25.943159 (+ 4082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:25.943164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:25.943269 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:25.943371 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:25.947369 (+ 3998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.949096 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.953324 (+ 4228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.954166 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.954248 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.963126 (+ 8878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:25.963133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:25.963140 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:25.963186 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:25.968607 (+ 5421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.969924 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.969932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.969937 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.970200 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:25.971154 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:25.971161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:25.971166 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:25.971735 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:25.971956 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:25.972614 (+ 658us) server_negotiation.cc:299] Negotiation successful
0808 12:57:25.972913 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5306,"thread_start_us":5166,"threads_started":1}
I20260808 12:57:25.975872 3209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.921355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35829 (local address 127.24.167.1:46193)
0808 12:57:25.939041 (+ 17686us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:25.939063 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:25.942826 (+ 3763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:25.944083 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:25.944095 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:25.945971 (+ 1876us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:25.947310 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.947326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.949338 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.949346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.953249 (+ 3903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.953264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.955286 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.955294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.955672 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.958251 (+ 2579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:25.958282 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:25.963213 (+ 4931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:25.968485 (+ 5272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.968497 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.968516 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.969511 (+ 995us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.970066 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.970070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.970074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.970491 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.971465 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:25.971473 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:25.972016 (+ 543us) client_negotiation.cc:769] Sending connection context
0808 12:57:25.973920 (+ 1904us) client_negotiation.cc:240] Negotiation successful
0808 12:57:25.974491 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17452,"mutex_wait_us":573,"spinlock_wait_cycles":107136,"thread_start_us":17554,"threads_started":1}
I20260808 12:57:25.976046 3211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.921507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:37779 (local address 127.24.167.1:42189)
0808 12:57:25.939899 (+ 18392us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:25.939930 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:25.942809 (+ 2879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:25.944082 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:25.944095 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:25.945734 (+ 1639us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:25.947135 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.947164 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.949330 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.949336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.953145 (+ 3809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.953160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.954833 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.954846 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.955689 (+ 843us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.956935 (+ 1246us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:25.956968 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:25.963625 (+ 6657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:25.968934 (+ 5309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.968940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.968945 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.969912 (+ 967us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.970503 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.970506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.970509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.970741 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.971467 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:25.971473 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:25.972016 (+ 543us) client_negotiation.cc:769] Sending connection context
0808 12:57:25.973546 (+ 1530us) client_negotiation.cc:240] Negotiation successful
0808 12:57:25.974130 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18245,"mutex_wait_us":803,"spinlock_wait_cycles":22144,"thread_start_us":14740,"threads_started":1}
I20260808 12:57:25.976519 3210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:25.921336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.1:47021)
0808 12:57:25.938858 (+ 17522us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:25.938927 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:25.942809 (+ 3882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:25.944082 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:25.944095 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:25.945448 (+ 1353us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:25.947135 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.947164 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.949330 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.949336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.953159 (+ 3823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:25.953168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:25.955000 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:25.955009 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:25.955687 (+ 678us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:25.956541 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:25.956578 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:25.962811 (+ 6233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:25.968816 (+ 6005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.968822 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.968827 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.969732 (+ 905us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.970425 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:25.970429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:25.970432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:25.970987 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:25.972062 (+ 1075us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:25.972068 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:25.972322 (+ 254us) client_negotiation.cc:769] Sending connection context
0808 12:57:25.973779 (+ 1457us) client_negotiation.cc:240] Negotiation successful
0808 12:57:25.974309 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15309,"mutex_wait_us":685,"spinlock_wait_cycles":37504,"thread_start_us":16485,"threads_started":1}
I20260808 12:57:25.978096 3199 heartbeater.cc:347] Connected to a master server at 127.24.167.62:34517
I20260808 12:57:25.978562 3199 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:25.978627 3200 heartbeater.cc:347] Connected to a master server at 127.24.167.61:37779
I20260808 12:57:25.978838 3200 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:25.978160 3198 heartbeater.cc:347] Connected to a master server at 127.24.167.60:35829
I20260808 12:57:25.979488 3198 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:25.979828 3199 heartbeater.cc:511] Master 127.24.167.62:34517 requested a full tablet report, sending...
I20260808 12:57:25.979977 3200 heartbeater.cc:511] Master 127.24.167.61:37779 requested a full tablet report, sending...
I20260808 12:57:25.980309 3198 heartbeater.cc:511] Master 127.24.167.60:35829 requested a full tablet report, sending...
I20260808 12:57:25.982113 1716 ts_manager.cc:194] Re-registered known tserver with Master: bc076c248e2541029f9460a5295055b9 (127.24.167.1:32969)
I20260808 12:57:25.983034 2901 ts_manager.cc:194] Registered new tserver with Master: bc076c248e2541029f9460a5295055b9 (127.24.167.1:32969)
I20260808 12:57:25.983108 2608 ts_manager.cc:194] Registered new tserver with Master: bc076c248e2541029f9460a5295055b9 (127.24.167.1:32969)
I20260808 12:57:25.983461 1716 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:47021
I20260808 12:57:26.053996 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 2006
I20260808 12:57:26.078065 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:34171
--local_ip_for_outbound_sockets=127.24.167.2
--tserver_master_addrs=127.24.167.62:34517
--webserver_port=34823
--webserver_interface=127.24.167.2
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.2.keytab
--principal=oryx/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.167.62:34517,127.24.167.61:37779
--tserver_master_addrs=127.24.167.62:34517,127.24.167.61:37779,127.24.167.60:35829 with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 12:57:26.097157 2987 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: NON_VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: true } } }
I20260808 12:57:26.097615 2987 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:26.099349 2987 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5: attempting to promote NON_VOTER e2bfd2b497ef40f9a6a450d9f11467fc to VOTER
I20260808 12:57:26.100207 2987 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } }
I20260808 12:57:26.102572 2619 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 FOLLOWER]: Refusing update from remote peer e2cb52625c1c4008a860a8dc90be27e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 12:57:26.102571 2916 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 1 LEARNER]: Refusing update from remote peer e2cb52625c1c4008a860a8dc90be27e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 12:57:26.103255 3216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 12:57:26.103758 2987 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 12:57:26.106884 2619 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e2bfd2b497ef40f9a6a450d9f11467fc (127.24.167.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } } }
I20260808 12:57:26.107476 2987 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e2bfd2b497ef40f9a6a450d9f11467fc (127.24.167.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } } }
I20260808 12:57:26.108804 2916 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e2bfd2b497ef40f9a6a450d9f11467fc (127.24.167.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } } }
I20260808 12:57:26.112393 3216 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } } }
I20260808 12:57:26.112680 3216 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:26.113046 3216 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } } }
I20260808 12:57:26.113284 3216 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:26.113646 3216 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } } }
I20260808 12:57:26.113868 3216 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:26.114491 3219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } } }
I20260808 12:57:26.114890 3219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 12:57:26.115272 2963 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2cb52625c1c4008a860a8dc90be27e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } } }
I20260808 12:57:26.115504 2963 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [sys.catalog]: This master's current role is: FOLLOWER
W20260808 12:57:26.425231 3214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:26.425563 3214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:26.425741 3214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:26.433502 3214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:26.433647 3214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:26.433847 3214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:57:26.444137 3214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.2.keytab
--principal=oryx/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:34171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=34823
--tserver_master_addrs=127.24.167.62:34517,127.24.167.61:37779,127.24.167.60:35829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:26.445572 3214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:26.447028 3214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:26.460613 3224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:26.460583 3225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:26.460824 3229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:26.461560 3214 server_base.cc:1034] running on GCE node
I20260808 12:57:26.462143 3214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:26.463225 3214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:26.464432 3214 hybrid_clock.cc:648] HybridClock initialized: now 1786193846464388 us; error 39 us; skew 500 ppm
I20260808 12:57:26.464787 3214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:26 dist-test-slave-tk9z krb5kdc[1644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193846, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:26.468245 3214 init.cc:377] Logged in from keytab as oryx/127.24.167.2@KRBTEST.COM (short username oryx)
I20260808 12:57:26.469619 3214 webserver.cc:460] Webserver started at http://127.24.167.2:34823/ using document root <none> and password file <none>
I20260808 12:57:26.470281 3214 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:26.470461 3214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:26.476150 3214 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 12:57:26.478912 3237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:26.480221 3214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 12:57:26.480417 3214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "927f604da71144bd8c483bee804ca845"
format_stamp: "Formatted at 2026-08-08 12:57:22 on dist-test-slave-tk9z"
server_key: "55219ed598cf97bf7d45b1c5d3d22e2b"
server_key_iv: "1832c3913eefaa2c239e751122d394bf"
server_key_version: "encryptionkey@0"
I20260808 12:57:26.481415 3214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:26.498149 3214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:26.502805 3214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:26.503132 3214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:26.504173 3214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:26.505564 3214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:26.505683 3214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:26.505828 3214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:26.505926 3214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:26.541790 3214 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:34171
I20260808 12:57:26.541841 3406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:34171 every 8 connection(s)
I20260808 12:57:26.543339 3214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:57:26.552003 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 3214
I20260808 12:57:26.552369 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 2199
Aug 08 12:57:26 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193846, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
Aug 08 12:57:26 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193846, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.61@KRBTEST.COM
Aug 08 12:57:26 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.167.3@KRBTEST.COM
Aug 08 12:57:26 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193846, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.60@KRBTEST.COM
I20260808 12:57:26.584311 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:26.546625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:49893 (local address 127.24.167.62:34517)
0808 12:57:26.557513 (+ 10888us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:26.557522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:26.557551 (+ 29us) server_negotiation.cc:407] Connection header received
0808 12:57:26.557639 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:26.557647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:26.557757 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:26.557912 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:26.562002 (+ 4090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:26.563567 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:26.567236 (+ 3669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:26.567795 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:26.567952 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:26.573848 (+ 5896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:26.573853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:26.573856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:26.573891 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:26.578726 (+ 4835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:26.580713 (+ 1987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:26.580720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:26.580726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:26.580905 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:26.582543 (+ 1638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:26.582551 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:26.582554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:26.582907 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:26.583062 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:26.583696 (+ 634us) server_negotiation.cc:299] Negotiation successful
0808 12:57:26.583918 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10705,"thread_start_us":11421,"threads_started":1}
I20260808 12:57:26.584370 3416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:26.546205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:38821 (local address 127.24.167.61:37779)
0808 12:57:26.558578 (+ 12373us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:26.558584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:26.561513 (+ 2929us) server_negotiation.cc:407] Connection header received
0808 12:57:26.561806 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:26.561812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:26.561966 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:26.562050 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:26.564587 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:26.566078 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:26.568738 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:26.569794 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:26.569899 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:26.575953 (+ 6054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:26.575960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:26.575974 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:26.576019 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:26.580537 (+ 4518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:26.581536 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:26.581544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:26.581550 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:26.581726 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:26.582166 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:26.582172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:26.582176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:26.582597 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:26.582747 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:26.583708 (+ 961us) server_negotiation.cc:299] Negotiation successful
0808 12:57:26.583965 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12246,"thread_start_us":12085,"threads_started":1}
I20260808 12:57:26.586058 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:26.547649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:37779 (local address 127.24.167.2:38821)
0808 12:57:26.561374 (+ 13725us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:26.561394 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:26.561640 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:26.562361 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:26.562365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:26.563606 (+ 1241us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:26.564373 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:26.564390 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:26.566360 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:26.566365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:26.568576 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:26.568591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:26.570280 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:26.570288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:26.570589 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:26.571301 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:26.571939 (+ 638us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:26.575686 (+ 3747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:26.580710 (+ 5024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:26.580716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:26.580722 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:26.581369 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:26.581884 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:26.581889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:26.581892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:26.582033 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:26.582859 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:26.582867 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:26.583324 (+ 457us) client_negotiation.cc:769] Sending connection context
0808 12:57:26.584586 (+ 1262us) client_negotiation.cc:240] Negotiation successful
0808 12:57:26.584898 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13529,"mutex_wait_us":1,"spinlock_wait_cycles":896,"thread_start_us":13105,"threads_started":1}
I20260808 12:57:26.586113 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:26.546919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.2:49893)
0808 12:57:26.548076 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:26.548132 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:26.551166 (+ 3034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:26.558811 (+ 7645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:26.558826 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:26.559747 (+ 921us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:26.561692 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:26.561721 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:26.563812 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:26.563817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:26.567047 (+ 3230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:26.567087 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:26.568048 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:26.568057 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:26.568504 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:26.569379 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:26.569419 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:26.573496 (+ 4077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:26.579668 (+ 6172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:26.579675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:26.579689 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:26.580322 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:26.581103 (+ 781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:26.581110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:26.581116 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:26.581457 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:26.583135 (+ 1678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:26.583138 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:26.583307 (+ 169us) client_negotiation.cc:769] Sending connection context
0808 12:57:26.583587 (+ 280us) client_negotiation.cc:240] Negotiation successful
0808 12:57:26.584153 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":27,"spinlock_wait_cycles":25856,"thread_start_us":1975,"threads_started":1}
I20260808 12:57:26.587191 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:26.554237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.24.167.3:39209)
0808 12:57:26.559104 (+ 4867us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:26.559113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:26.559699 (+ 586us) server_negotiation.cc:407] Connection header received
0808 12:57:26.559864 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:26.559877 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:26.560168 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:26.561086 (+ 918us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:26.563885 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:26.566115 (+ 2230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:26.568521 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:26.569458 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:26.569583 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:26.578160 (+ 8577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:26.578178 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:26.578211 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:26.578294 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:26.582917 (+ 4623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:26.583772 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:26.583780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:26.583783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:26.583964 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:26.584431 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:26.584436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:26.584439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:26.584860 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:26.585167 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:26.586477 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0808 12:57:26.586767 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4652,"thread_start_us":4484,"threads_started":1}
I20260808 12:57:26.587749 3410 heartbeater.cc:347] Connected to a master server at 127.24.167.61:37779
I20260808 12:57:26.588086 3410 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:26.588742 2380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:26.589293 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:26.548043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35829 (local address 127.24.167.2:58147)
0808 12:57:26.561477 (+ 13434us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:26.561499 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:26.561713 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:26.562345 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:26.562351 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:26.563847 (+ 1496us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:26.564752 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:26.564768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:26.566813 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:26.566820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:26.569641 (+ 2821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:26.569656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:26.571389 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:26.571397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:26.571551 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:26.573337 (+ 1786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:26.573362 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:26.578630 (+ 5268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:26.582487 (+ 3857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:26.582494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:26.582497 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:26.583010 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:26.586858 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:26.586862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:26.586866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:26.587036 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:26.588294 (+ 1258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:26.588298 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:26.588413 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 12:57:26.588557 (+ 144us) client_negotiation.cc:240] Negotiation successful
0808 12:57:26.588845 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13245,"thread_start_us":12252,"threads_started":1}
I20260808 12:57:26.589272 3410 heartbeater.cc:511] Master 127.24.167.61:37779 requested a full tablet report, sending...
I20260808 12:57:26.587723 3409 heartbeater.cc:347] Connected to a master server at 127.24.167.62:34517
I20260808 12:57:26.589514 3409 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:26.589665 3417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:26.546205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:58147 (local address 127.24.167.60:35829)
0808 12:57:26.546794 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:26.546800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:26.561599 (+ 14799us) server_negotiation.cc:407] Connection header received
0808 12:57:26.561879 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:26.561887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:26.561983 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:26.562071 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:26.564971 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:26.566545 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:26.569822 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:26.570758 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:26.570876 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:26.578924 (+ 8048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:26.578931 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:26.578935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:26.578971 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:26.582314 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:26.586031 (+ 3717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:26.586040 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:26.586043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:26.586698 (+ 655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:26.587173 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:26.587178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:26.587181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:26.588061 (+ 880us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:26.588200 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:26.588687 (+ 487us) server_negotiation.cc:299] Negotiation successful
0808 12:57:26.588996 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":564,"threads_started":1}
I20260808 12:57:26.590523 3409 heartbeater.cc:511] Master 127.24.167.62:34517 requested a full tablet report, sending...
I20260808 12:57:26.591200 3407 heartbeater.cc:347] Connected to a master server at 127.24.167.60:35829
I20260808 12:57:26.591349 3407 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:26.591692 3407 heartbeater.cc:511] Master 127.24.167.60:35829 requested a full tablet report, sending...
I20260808 12:57:26.591928 2608 ts_manager.cc:194] Registered new tserver with Master: 927f604da71144bd8c483bee804ca845 (127.24.167.2:34171)
I20260808 12:57:26.592806 2901 ts_manager.cc:194] Registered new tserver with Master: 927f604da71144bd8c483bee804ca845 (127.24.167.2:34171)
I20260808 12:57:26.593155 1716 ts_manager.cc:194] Re-registered known tserver with Master: 927f604da71144bd8c483bee804ca845 (127.24.167.2:34171)
I20260808 12:57:26.593992 1716 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:49893
I20260808 12:57:26.683389 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 2199
I20260808 12:57:26.706117 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:39209
--local_ip_for_outbound_sockets=127.24.167.3
--tserver_master_addrs=127.24.167.62:34517
--webserver_port=40679
--webserver_interface=127.24.167.3
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.3.keytab
--principal=oryx/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:39739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.167.62:34517,127.24.167.61:37779
--tserver_master_addrs=127.24.167.62:34517,127.24.167.61:37779,127.24.167.60:35829 with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 12:57:26.986135 3199 heartbeater.cc:503] Master 127.24.167.62:34517 was elected leader, sending a full tablet report...
W20260808 12:57:27.058167 3430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:27.058485 3430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:27.058584 3430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:27.065975 3430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:27.066082 3430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:27.066193 3430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:57:27.076037 3430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:39739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.3.keytab
--principal=oryx/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:39209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=40679
--tserver_master_addrs=127.24.167.62:34517,127.24.167.61:37779,127.24.167.60:35829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:27.077312 3430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:27.078722 3430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:27.092664 3437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:27.092727 3440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:27.092689 3442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:27.094286 3430 server_base.cc:1034] running on GCE node
I20260808 12:57:27.094954 3430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:27.096071 3430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:27.097306 3430 hybrid_clock.cc:648] HybridClock initialized: now 1786193847097253 us; error 48 us; skew 500 ppm
I20260808 12:57:27.097647 3430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:27 dist-test-slave-tk9z krb5kdc[1644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193847, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:27.100878 3430 init.cc:377] Logged in from keytab as oryx/127.24.167.3@KRBTEST.COM (short username oryx)
I20260808 12:57:27.102304 3430 webserver.cc:460] Webserver started at http://127.24.167.3:40679/ using document root <none> and password file <none>
I20260808 12:57:27.103011 3430 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:27.103128 3430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:27.108469 3430 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260808 12:57:27.111773 3453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:27.113480 3430 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260808 12:57:27.113622 3430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "62ff2f69928b42d3b99721fa260c0c5d"
format_stamp: "Formatted at 2026-08-08 12:57:23 on dist-test-slave-tk9z"
server_key: "7639de76856eca476791dfc438474fec"
server_key_iv: "398b9fbff32757653061e45a59f63b1a"
server_key_version: "encryptionkey@0"
I20260808 12:57:27.114719 3430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:27.125336 3430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:27.130790 3430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:27.131136 3430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:27.132509 3430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:27.134581 3430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:27.134704 3430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:27.134819 3430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:27.134896 3430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:27.170774 3430 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:39209
I20260808 12:57:27.170872 3620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:39209 every 8 connection(s)
I20260808 12:57:27.172254 3430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:57:27.181902 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 3430
Aug 08 12:57:27 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193847, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
Aug 08 12:57:27 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193847, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.60@KRBTEST.COM
Aug 08 12:57:27 dist-test-slave-tk9z krb5kdc[1644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193847, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.61@KRBTEST.COM
I20260808 12:57:27.216646 3628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.175251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:44095 (local address 127.24.167.62:34517)
0808 12:57:27.175720 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.175725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.177638 (+ 1913us) server_negotiation.cc:407] Connection header received
0808 12:57:27.180154 (+ 2516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.180160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.180262 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.180385 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:27.183124 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.183966 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.186041 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.186691 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.186825 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.194181 (+ 7356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:27.194192 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:27.194200 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:27.194255 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:27.211303 (+ 17048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.212298 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.212307 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.212312 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.212594 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.213335 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.213342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.213346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.213859 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:27.214118 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.215869 (+ 1751us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.216166 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":248,"threads_started":1}
I20260808 12:57:27.218364 3635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.176570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.3:44095)
0808 12:57:27.177304 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:27.177356 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:27.179875 (+ 2519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:27.180632 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:27.180647 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:27.181657 (+ 1010us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:27.182955 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.182980 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.184105 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.184113 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.185895 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.185915 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.186929 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.186937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.187228 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.187998 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:27.188030 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:27.193824 (+ 5794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:27.211471 (+ 17647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:27.211480 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:27.211486 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:27.212120 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:27.212751 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:27.212759 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:27.212765 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:27.213114 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:27.214080 (+ 966us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:27.214093 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:27.214862 (+ 769us) client_negotiation.cc:769] Sending connection context
0808 12:57:27.215324 (+ 462us) client_negotiation.cc:240] Negotiation successful
0808 12:57:27.216251 (+ 927us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":209,"spinlock_wait_cycles":2816,"thread_start_us":432,"threads_started":1}
I20260808 12:57:27.222409 3622 heartbeater.cc:347] Connected to a master server at 127.24.167.62:34517
I20260808 12:57:27.222797 3622 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:27.222931 3638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.178968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35829 (local address 127.24.167.3:35737)
0808 12:57:27.179651 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:27.179668 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:27.179876 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:27.180633 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:27.180643 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:27.181626 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:27.182952 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.182982 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.184466 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.184471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.187242 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.187257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.188820 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.188831 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.189046 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.189801 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:27.189831 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:27.193985 (+ 4154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:27.209496 (+ 15511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:27.209514 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:27.209546 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:27.210951 (+ 1405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:27.219529 (+ 8578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:27.219538 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:27.219545 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:27.219776 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:27.220782 (+ 1006us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:27.220790 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:27.221062 (+ 272us) client_negotiation.cc:769] Sending connection context
0808 12:57:27.221257 (+ 195us) client_negotiation.cc:240] Negotiation successful
0808 12:57:27.222409 (+ 1152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":998,"spinlock_wait_cycles":73088,"thread_start_us":543,"threads_started":1}
I20260808 12:57:27.222993 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.175259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:45945 (local address 127.24.167.61:37779)
0808 12:57:27.178809 (+ 3550us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.178825 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.178847 (+ 22us) server_negotiation.cc:407] Connection header received
0808 12:57:27.180653 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.180658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.180746 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.180871 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:27.184166 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.185278 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.188314 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.189012 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.189130 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.201187 (+ 12057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:27.201199 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:27.201206 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:27.201265 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:27.208746 (+ 7481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.219828 (+ 11082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.219837 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.219842 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.220054 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.220846 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.220852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.220856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.221277 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:27.221416 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.222219 (+ 803us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.222541 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3431,"thread_start_us":3330,"threads_started":1}
I20260808 12:57:27.224341 3637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.178932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:35737 (local address 127.24.167.60:35829)
0808 12:57:27.179421 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.179427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.179760 (+ 333us) server_negotiation.cc:407] Connection header received
0808 12:57:27.180176 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.180184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.180264 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.180381 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:27.183175 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.184290 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.187419 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.188446 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.188575 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.194258 (+ 5683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:27.194267 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:27.194274 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:27.194324 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:27.200978 (+ 6654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.211130 (+ 10152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.211139 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.211146 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.218926 (+ 7780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.219970 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.219979 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.219985 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.220500 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:27.220699 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.223237 (+ 2538us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.223587 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":263,"threads_started":1}
I20260808 12:57:27.224606 3622 heartbeater.cc:511] Master 127.24.167.62:34517 requested a full tablet report, sending...
I20260808 12:57:27.225675 3621 heartbeater.cc:347] Connected to a master server at 127.24.167.60:35829
I20260808 12:57:27.225721 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.175913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:37779 (local address 127.24.167.3:45945)
0808 12:57:27.177694 (+ 1781us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:27.177710 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:27.179875 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:27.180912 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:27.180918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:27.181657 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:27.182954 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.182980 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.185443 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.185449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.188130 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.188147 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.189480 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.189487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.189751 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.196115 (+ 6364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:27.196148 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:27.200298 (+ 4150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:27.214811 (+ 14513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:27.214816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:27.214819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:27.215532 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:27.220256 (+ 4724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:27.220277 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:27.220281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:27.220663 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:27.221562 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:27.221567 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:27.221812 (+ 245us) client_negotiation.cc:769] Sending connection context
0808 12:57:27.223250 (+ 1438us) client_negotiation.cc:240] Negotiation successful
0808 12:57:27.223572 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":149,"spinlock_wait_cycles":2816,"thread_start_us":709,"threads_started":1}
I20260808 12:57:27.225863 3621 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:27.226898 3621 heartbeater.cc:511] Master 127.24.167.60:35829 requested a full tablet report, sending...
I20260808 12:57:27.227011 3623 heartbeater.cc:347] Connected to a master server at 127.24.167.61:37779
I20260808 12:57:27.227174 3623 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:27.227805 3623 heartbeater.cc:511] Master 127.24.167.61:37779 requested a full tablet report, sending...
I20260808 12:57:27.228860 2901 ts_manager.cc:194] Registered new tserver with Master: 62ff2f69928b42d3b99721fa260c0c5d (127.24.167.3:39209)
I20260808 12:57:27.229302 1716 ts_manager.cc:194] Re-registered known tserver with Master: 62ff2f69928b42d3b99721fa260c0c5d (127.24.167.3:39209)
I20260808 12:57:27.229523 2608 ts_manager.cc:194] Registered new tserver with Master: 62ff2f69928b42d3b99721fa260c0c5d (127.24.167.3:39209)
I20260808 12:57:27.230703 1716 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:44095
I20260808 12:57:27.244626 3628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.190723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.24.167.62:34517)
0808 12:57:27.217014 (+ 26291us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.217020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.217037 (+ 17us) server_negotiation.cc:407] Connection header received
0808 12:57:27.217087 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.217091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.217167 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.218497 (+ 1330us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:27.223432 (+ 4935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.225895 (+ 2463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.228834 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.230460 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.230953 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.236134 (+ 5181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:27.236143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:27.236150 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:27.236210 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:27.241195 (+ 4985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.241918 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.241924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.241928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.242135 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.242710 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.242716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.242719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.243173 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:27.243547 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.243920 (+ 373us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.244185 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26167,"thread_start_us":34638,"threads_started":1}
I20260808 12:57:27.248823 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.201550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59104 (local address 127.24.167.61:37779)
0808 12:57:27.206300 (+ 4750us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.206308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.211184 (+ 4876us) server_negotiation.cc:407] Connection header received
0808 12:57:27.211250 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.211254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.211330 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.211428 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:27.215079 (+ 3651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.217131 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.233101 (+ 15970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.233854 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.233987 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.238856 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:27.238868 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:27.238871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:27.238910 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:27.243879 (+ 4969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.245216 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.245222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.245227 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.245569 (+ 342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.246260 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.246278 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.246298 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.246855 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:27.247232 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.247885 (+ 653us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.248165 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4603,"thread_start_us":4471,"threads_started":1}
I20260808 12:57:27.251152 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.189027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.24.167.60:35829)
0808 12:57:27.217526 (+ 28499us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.217533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.217552 (+ 19us) server_negotiation.cc:407] Connection header received
0808 12:57:27.217611 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.217616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.217691 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.217796 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:27.232810 (+ 15014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.234417 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.236840 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.237440 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.237581 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.241298 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:27.241310 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:27.241319 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:27.241370 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:27.246665 (+ 5295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.247739 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.247744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.247747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.247997 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.248674 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.248680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.248683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.249201 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:27.249515 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.250405 (+ 890us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.250715 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28336,"thread_start_us":11839,"threads_started":1}
I20260808 12:57:27.249634 1716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:57:27.258944 1716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:57:27.313753 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.289992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.24.167.1:32969)
0808 12:57:27.303358 (+ 13366us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.303379 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.303417 (+ 38us) server_negotiation.cc:407] Connection header received
0808 12:57:27.303581 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.303592 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.303872 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.304110 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:27.305615 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.307160 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.310663 (+ 3503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.312488 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.312640 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.312906 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.313177 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.313357 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13036,"thread_start_us":14749,"threads_started":1}
I20260808 12:57:27.314213 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.289640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:32969 (local address 127.0.0.1:36780)
0808 12:57:27.299971 (+ 10331us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:27.299994 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:27.300167 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:27.304417 (+ 4250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:27.304422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:27.304447 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:27.305323 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.305360 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.307288 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.307294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.310447 (+ 3153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.310461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.312719 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.312727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.312970 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.313004 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 12:57:27.313131 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 12:57:27.313369 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10167,"mutex_wait_us":120,"thread_start_us":9958,"threads_started":1}
I20260808 12:57:27.314528 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.283193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:34171 (local address 127.0.0.1:54830)
0808 12:57:27.297186 (+ 13993us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:27.297212 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:27.297356 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:27.304571 (+ 7215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:27.304576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:27.304605 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:27.305713 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.305730 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.307980 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.307985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.311312 (+ 3327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.311327 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.313140 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.313148 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.313416 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.313454 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 12:57:27.313628 (+ 174us) client_negotiation.cc:240] Negotiation successful
0808 12:57:27.313769 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13804,"mutex_wait_us":38,"thread_start_us":14260,"threads_started":1}
I20260808 12:57:27.315135 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.283430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54830 (local address 127.24.167.2:34171)
0808 12:57:27.302900 (+ 19470us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.302909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.302939 (+ 30us) server_negotiation.cc:407] Connection header received
0808 12:57:27.303033 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.303040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.303252 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.303489 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:27.305960 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.307809 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.311599 (+ 3790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.312878 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.313081 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.314223 (+ 1142us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.314584 (+ 361us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.314914 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19312,"thread_start_us":19209,"threads_started":1}
I20260808 12:57:27.318287 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.282758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:39209 (local address 127.0.0.1:45250)
0808 12:57:27.290549 (+ 7791us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:27.290574 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:27.290770 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:27.305430 (+ 14660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:27.305437 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:27.305474 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:27.306343 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.306358 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.308600 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.308606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.311382 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.311395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.316896 (+ 5501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.316903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.317284 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.317317 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 12:57:27.317474 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 12:57:27.317661 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6613,"mutex_wait_us":138,"thread_start_us":6443,"threads_started":1}
I20260808 12:57:27.318822 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.283390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45250 (local address 127.24.167.3:39209)
0808 12:57:27.304675 (+ 21285us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.304684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.304760 (+ 76us) server_negotiation.cc:407] Connection header received
0808 12:57:27.304868 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.304875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.305156 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.305414 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:27.306662 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.308446 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.313497 (+ 5051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.316672 (+ 3175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.316823 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.317133 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.317584 (+ 451us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.317859 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21060,"thread_start_us":20779,"threads_started":1}
I20260808 12:57:27.319415 3315 tablet_service.cc:1467] Processing CreateTablet for tablet 57ca2eb3580f4777995c5f54ac69c983 (DEFAULT_TABLE table=test-table [id=25233bccf0524aadafc2ad833ae4665b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:27.319453 3090 tablet_service.cc:1467] Processing CreateTablet for tablet 57ca2eb3580f4777995c5f54ac69c983 (DEFAULT_TABLE table=test-table [id=25233bccf0524aadafc2ad833ae4665b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:27.321213 3315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ca2eb3580f4777995c5f54ac69c983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:27.321394 3090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ca2eb3580f4777995c5f54ac69c983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:27.322563 3526 tablet_service.cc:1467] Processing CreateTablet for tablet 57ca2eb3580f4777995c5f54ac69c983 (DEFAULT_TABLE table=test-table [id=25233bccf0524aadafc2ad833ae4665b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:27.323812 3526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ca2eb3580f4777995c5f54ac69c983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:27.338124 3681 tablet_bootstrap.cc:492] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d: Bootstrap starting.
I20260808 12:57:27.340502 3688 tablet_bootstrap.cc:492] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9: Bootstrap starting.
I20260808 12:57:27.343218 3681 tablet_bootstrap.cc:654] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:27.343454 3684 tablet_bootstrap.cc:492] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845: Bootstrap starting.
I20260808 12:57:27.345041 3688 tablet_bootstrap.cc:654] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:27.347095 3681 log.cc:824] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:27.347055 3688 log.cc:824] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:27.348264 3684 tablet_bootstrap.cc:654] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:27.350045 3684 log.cc:824] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:27.352219 3688 tablet_bootstrap.cc:492] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9: No bootstrap required, opened a new log
I20260808 12:57:27.352384 3681 tablet_bootstrap.cc:492] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d: No bootstrap required, opened a new log
I20260808 12:57:27.352562 3688 ts_tablet_manager.cc:1397] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260808 12:57:27.352779 3681 ts_tablet_manager.cc:1397] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20260808 12:57:27.354112 3684 tablet_bootstrap.cc:492] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845: No bootstrap required, opened a new log
I20260808 12:57:27.354528 3684 ts_tablet_manager.cc:1397] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260808 12:57:27.362950 3681 raft_consensus.cc:348] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } }
I20260808 12:57:27.363400 3681 raft_consensus.cc:374] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:27.363519 3681 raft_consensus.cc:729] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ff2f69928b42d3b99721fa260c0c5d, State: Initialized, Role: FOLLOWER
I20260808 12:57:27.364318 3681 consensus_queue.cc:260] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } }
I20260808 12:57:27.365746 3688 raft_consensus.cc:348] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } }
I20260808 12:57:27.366405 3688 raft_consensus.cc:374] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:27.366619 3688 raft_consensus.cc:729] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc076c248e2541029f9460a5295055b9, State: Initialized, Role: FOLLOWER
I20260808 12:57:27.366870 3681 ts_tablet_manager.cc:1428] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260808 12:57:27.367177 3622 heartbeater.cc:503] Master 127.24.167.62:34517 was elected leader, sending a full tablet report...
I20260808 12:57:27.367451 3688 consensus_queue.cc:260] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } }
I20260808 12:57:27.369551 3684 raft_consensus.cc:348] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } }
I20260808 12:57:27.370193 3684 raft_consensus.cc:374] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:27.370393 3684 raft_consensus.cc:729] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927f604da71144bd8c483bee804ca845, State: Initialized, Role: FOLLOWER
I20260808 12:57:27.372061 3688 ts_tablet_manager.cc:1428] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9: Time spent starting tablet: real 0.019s user 0.014s sys 0.006s
I20260808 12:57:27.373142 3684 consensus_queue.cc:260] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } }
I20260808 12:57:27.377511 3684 ts_tablet_manager.cc:1428] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845: Time spent starting tablet: real 0.023s user 0.007s sys 0.012s
I20260808 12:57:27.377995 3409 heartbeater.cc:503] Master 127.24.167.62:34517 was elected leader, sending a full tablet report...
W20260808 12:57:27.417235 3201 tablet.cc:2369] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 12:57:27.425428 3627 tablet.cc:2369] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:57:27.532086 3694 raft_consensus.cc:482] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:57:27.532518 3694 raft_consensus.cc:504] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } }
I20260808 12:57:27.534801 3694 leader_election.cc:288] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62ff2f69928b42d3b99721fa260c0c5d (127.24.167.3:39209), 927f604da71144bd8c483bee804ca845 (127.24.167.2:34171)
I20260808 12:57:27.544436 3695 raft_consensus.cc:482] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:57:27.544792 3695 raft_consensus.cc:504] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } }
I20260808 12:57:27.545667 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.535023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:34171 (local address 127.24.167.1:48027)
0808 12:57:27.537002 (+ 1979us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:27.537017 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:27.537277 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:27.537814 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:27.537817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:27.537838 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:27.538484 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.538501 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.539645 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.539651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.542032 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.542045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.544042 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.544048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.544293 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.544328 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 12:57:27.544822 (+ 494us) client_negotiation.cc:240] Negotiation successful
0808 12:57:27.544971 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1850,"mutex_wait_us":24,"spinlock_wait_cycles":2944,"thread_start_us":1442,"threads_started":1}
I20260808 12:57:27.545709 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.535034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:50825 (local address 127.24.167.3:39209)
0808 12:57:27.535326 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.535336 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.537097 (+ 1761us) server_negotiation.cc:407] Connection header received
0808 12:57:27.537348 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.537359 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.537446 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.537532 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:27.538530 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.539512 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.542957 (+ 3445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.544290 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.544471 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.544547 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.545231 (+ 684us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.545361 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 12:57:27.546635 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.534930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:39209 (local address 127.24.167.1:50825)
0808 12:57:27.536950 (+ 2020us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:27.536970 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:27.537165 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:27.537682 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:27.537687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:27.537724 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:27.538317 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.538328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.539719 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.539723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.542789 (+ 3066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.542804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.544708 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.544717 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.544960 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.545003 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 12:57:27.545946 (+ 943us) client_negotiation.cc:240] Negotiation successful
0808 12:57:27.546101 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1830,"spinlock_wait_cycles":6912,"thread_start_us":1493,"threads_started":1}
W20260808 12:57:27.546869 3411 tablet.cc:2369] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:57:27.547312 3695 leader_election.cc:288] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62ff2f69928b42d3b99721fa260c0c5d (127.24.167.3:39209), bc076c248e2541029f9460a5295055b9 (127.24.167.1:32969)
I20260808 12:57:27.548792 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.535424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:48027 (local address 127.24.167.2:34171)
0808 12:57:27.535668 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.535676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.537133 (+ 1457us) server_negotiation.cc:407] Connection header received
0808 12:57:27.537481 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.537485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.537681 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.537781 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:27.538675 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.539465 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.542259 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.543794 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.543989 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.544081 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.544504 (+ 423us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.544654 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 12:57:27.549024 3550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ca2eb3580f4777995c5f54ac69c983" candidate_uuid: "bc076c248e2541029f9460a5295055b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62ff2f69928b42d3b99721fa260c0c5d" is_pre_election: true
I20260808 12:57:27.549764 3550 raft_consensus.cc:2454] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc076c248e2541029f9460a5295055b9 in term 0.
I20260808 12:57:27.550513 3015 leader_election.cc:302] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62ff2f69928b42d3b99721fa260c0c5d, bc076c248e2541029f9460a5295055b9; no voters:
I20260808 12:57:27.550944 3694 raft_consensus.cc:2781] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:57:27.551167 3694 raft_consensus.cc:482] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:57:27.551324 3694 raft_consensus.cc:3037] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:27.555519 3694 raft_consensus.cc:504] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } }
I20260808 12:57:27.556633 3550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ca2eb3580f4777995c5f54ac69c983" candidate_uuid: "bc076c248e2541029f9460a5295055b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62ff2f69928b42d3b99721fa260c0c5d"
I20260808 12:57:27.556840 3550 raft_consensus.cc:3037] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:27.560904 3694 leader_election.cc:288] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [CANDIDATE]: Term 1 election: Requested vote from peers 62ff2f69928b42d3b99721fa260c0c5d (127.24.167.3:39209), 927f604da71144bd8c483bee804ca845 (127.24.167.2:34171)
I20260808 12:57:27.561429 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.547957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:49719 (local address 127.24.167.3:39209)
0808 12:57:27.548237 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.548243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.550433 (+ 2190us) server_negotiation.cc:407] Connection header received
0808 12:57:27.550837 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.550846 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.551001 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.551177 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:27.552726 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.553981 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.558541 (+ 4560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.560017 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.560138 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.560208 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.560854 (+ 646us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.561018 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 12:57:27.561517 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.547468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:39209 (local address 127.24.167.2:49719)
0808 12:57:27.550306 (+ 2838us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:27.550327 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:27.550546 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:27.551555 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:27.551563 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:27.551602 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:27.552540 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.552555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.554223 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.554228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.557492 (+ 3264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.557527 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.560496 (+ 2969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.560502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.560674 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.560708 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 12:57:27.560811 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 12:57:27.560957 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2604,"thread_start_us":2058,"threads_started":1}
I20260808 12:57:27.561458 3550 raft_consensus.cc:2454] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc076c248e2541029f9460a5295055b9 in term 1.
I20260808 12:57:27.561940 3015 leader_election.cc:302] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62ff2f69928b42d3b99721fa260c0c5d, bc076c248e2541029f9460a5295055b9; no voters:
I20260808 12:57:27.562314 3694 raft_consensus.cc:2781] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:27.562425 3341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ca2eb3580f4777995c5f54ac69c983" candidate_uuid: "bc076c248e2541029f9460a5295055b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927f604da71144bd8c483bee804ca845" is_pre_election: true
I20260808 12:57:27.563063 3341 raft_consensus.cc:2454] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc076c248e2541029f9460a5295055b9 in term 0.
I20260808 12:57:27.563596 3694 raft_consensus.cc:686] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [term 1 LEADER]: Becoming Leader. State: Replica: bc076c248e2541029f9460a5295055b9, State: Running, Role: LEADER
I20260808 12:57:27.564612 3694 consensus_queue.cc:237] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } }
I20260808 12:57:27.566794 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.550306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:49993 (local address 127.24.167.1:32969)
0808 12:57:27.550542 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.550548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.558288 (+ 7740us) server_negotiation.cc:407] Connection header received
0808 12:57:27.558508 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.558514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.558582 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.558668 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:27.559510 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.560706 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.564019 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.565277 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.565386 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.565444 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.566148 (+ 704us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.566276 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 12:57:27.566829 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.550021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:32969 (local address 127.24.167.2:49993)
0808 12:57:27.558150 (+ 8129us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:27.558170 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:27.558346 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:27.558810 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:27.558815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:27.558839 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:27.559330 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.559341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.560931 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.560937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.563840 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:27.563856 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.565798 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.565805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:27.565980 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.566009 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 12:57:27.566123 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 12:57:27.566259 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7961,"mutex_wait_us":536,"thread_start_us":7200,"threads_started":1}
I20260808 12:57:27.566886 3341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ca2eb3580f4777995c5f54ac69c983" candidate_uuid: "bc076c248e2541029f9460a5295055b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927f604da71144bd8c483bee804ca845"
I20260808 12:57:27.567174 3341 raft_consensus.cc:3037] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:27.568305 3550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ca2eb3580f4777995c5f54ac69c983" candidate_uuid: "927f604da71144bd8c483bee804ca845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62ff2f69928b42d3b99721fa260c0c5d" is_pre_election: true
I20260808 12:57:27.568727 3550 raft_consensus.cc:2379] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 927f604da71144bd8c483bee804ca845 in current term 1: Already voted for candidate bc076c248e2541029f9460a5295055b9 in this term.
I20260808 12:57:27.568781 3124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ca2eb3580f4777995c5f54ac69c983" candidate_uuid: "927f604da71144bd8c483bee804ca845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc076c248e2541029f9460a5295055b9" is_pre_election: true
I20260808 12:57:27.570333 3243 leader_election.cc:302] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 927f604da71144bd8c483bee804ca845; no voters: 62ff2f69928b42d3b99721fa260c0c5d, bc076c248e2541029f9460a5295055b9
I20260808 12:57:27.572206 3341 raft_consensus.cc:2454] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc076c248e2541029f9460a5295055b9 in term 1.
I20260808 12:57:27.572479 3695 raft_consensus.cc:2726] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 12:57:27.576018 1716 catalog_manager.cc:5495] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc076c248e2541029f9460a5295055b9 (127.24.167.1). New cstate: current_term: 1 leader_uuid: "bc076c248e2541029f9460a5295055b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } health_report { overall_health: HEALTHY } } }
I20260808 12:57:27.661765 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.655296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.24.167.1:32969)
0808 12:57:27.655478 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.655484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.655507 (+ 23us) server_negotiation.cc:407] Connection header received
0808 12:57:27.655738 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.655745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.655879 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.656024 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:57:27.656474 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.657416 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.659075 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.659862 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.660046 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.660545 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:57:27.661152 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:57:27.661269 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.661375 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.661451 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 12:57:27.672559 3550 raft_consensus.cc:1261] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d [term 1 FOLLOWER]: Refusing update from remote peer bc076c248e2541029f9460a5295055b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:27.672642 3341 raft_consensus.cc:1261] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845 [term 1 FOLLOWER]: Refusing update from remote peer bc076c248e2541029f9460a5295055b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:27.673895 3702 consensus_queue.cc:1035] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:27.674319 3694 consensus_queue.cc:1035] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:27.691098 3707 mvcc.cc:205] Tried to move back new op lower bound from 7316250000053489664 to 7316249999630987264. Current Snapshot: MvccSnapshot[applied={T|T < 7316250000053489664}]
I20260808 12:57:27.712450 1716 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:27.712883 1716 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:27.721076 1716 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2cb52625c1c4008a860a8dc90be27e5: Sending DeleteTablet for 3 replicas of tablet 57ca2eb3580f4777995c5f54ac69c983
I20260808 12:57:27.722502 3526 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ca2eb3580f4777995c5f54ac69c983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:27 UTC) from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:45250
I20260808 12:57:27.723237 3315 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ca2eb3580f4777995c5f54ac69c983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:27 UTC) from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:54830
I20260808 12:57:27.723685 3725 tablet_replica.cc:330] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d: stopping tablet replica
I20260808 12:57:27.724301 3725 raft_consensus.cc:2229] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:27.724864 3725 raft_consensus.cc:2258] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:27.727751 3725 ts_tablet_manager.cc:1905] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:27.730536 3090 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ca2eb3580f4777995c5f54ac69c983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:27 UTC) from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:36780
I20260808 12:57:27.738907 3725 ts_tablet_manager.cc:1918] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:27.739203 3725 log.cc:1196] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/57ca2eb3580f4777995c5f54ac69c983
I20260808 12:57:27.740223 3725 ts_tablet_manager.cc:1939] T 57ca2eb3580f4777995c5f54ac69c983 P 62ff2f69928b42d3b99721fa260c0c5d: Deleting consensus metadata
I20260808 12:57:27.742828 1697 catalog_manager.cc:4841] TS 62ff2f69928b42d3b99721fa260c0c5d (127.24.167.3:39209): tablet 57ca2eb3580f4777995c5f54ac69c983 (table test-table [id=25233bccf0524aadafc2ad833ae4665b]) successfully deleted
I20260808 12:57:27.743943 3729 tablet_replica.cc:330] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9: stopping tablet replica
I20260808 12:57:27.744627 3729 raft_consensus.cc:2229] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:57:27.745726 3729 raft_consensus.cc:2258] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:27.755106 3727 tablet_replica.cc:330] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845: stopping tablet replica
I20260808 12:57:27.755743 3727 raft_consensus.cc:2229] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:27.756381 3727 raft_consensus.cc:2258] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:27.756865 3729 ts_tablet_manager.cc:1905] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:27.758605 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 1660
I20260808 12:57:27.761175 1771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:27.765559 3727 ts_tablet_manager.cc:1905] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:27.767107 3729 ts_tablet_manager.cc:1918] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:27.767460 3729 log.cc:1196] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/57ca2eb3580f4777995c5f54ac69c983
I20260808 12:57:27.769814 3729 ts_tablet_manager.cc:1939] T 57ca2eb3580f4777995c5f54ac69c983 P bc076c248e2541029f9460a5295055b9: Deleting consensus metadata
I20260808 12:57:27.775481 3727 ts_tablet_manager.cc:1918] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:27.775705 3727 log.cc:1196] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/57ca2eb3580f4777995c5f54ac69c983
I20260808 12:57:27.776610 3727 ts_tablet_manager.cc:1939] T 57ca2eb3580f4777995c5f54ac69c983 P 927f604da71144bd8c483bee804ca845: Deleting consensus metadata
I20260808 12:57:27.785203 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.725110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.24.167.61:37779)
0808 12:57:27.725336 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.725342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.757359 (+ 32017us) server_negotiation.cc:407] Connection header received
0808 12:57:27.760030 (+ 2671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.760038 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.760142 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.760237 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:27.763226 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.764590 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.767236 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.767919 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.768074 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.778238 (+ 10164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:27.778245 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:27.778249 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:27.778298 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:27.781633 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.782701 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.782707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.782710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.782976 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.783472 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.783478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.783481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.783899 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:27.784210 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.784522 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.784829 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 12:57:27.800616 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.757549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.24.167.60:35829)
0808 12:57:27.759709 (+ 2160us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.759714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.785406 (+ 25692us) server_negotiation.cc:407] Connection header received
0808 12:57:27.785699 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.785705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.785827 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.785938 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:27.787888 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.788737 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.790236 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.790943 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.791097 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:27.793996 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:27.794007 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:27.794010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:27.794046 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:27.797536 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.798355 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.798363 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.798425 (+ 62us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.798569 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:27.798992 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:27.798998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:27.799001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:27.799283 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:27.799604 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:27.799995 (+ 391us) server_negotiation.cc:299] Negotiation successful
0808 12:57:27.800230 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1978,"thread_start_us":1777,"threads_started":1}
I20260808 12:57:27.943745 1697 catalog_manager.cc:4841] TS bc076c248e2541029f9460a5295055b9 (127.24.167.1:32969): tablet 57ca2eb3580f4777995c5f54ac69c983 (table test-table [id=25233bccf0524aadafc2ad833ae4665b]) successfully deleted
I20260808 12:57:27.944751 1686 catalog_manager.cc:4841] TS 927f604da71144bd8c483bee804ca845 (127.24.167.2:34171): tablet 57ca2eb3580f4777995c5f54ac69c983 (table test-table [id=25233bccf0524aadafc2ad833ae4665b]) successfully deleted
I20260808 12:57:27.946156 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 1660
I20260808 12:57:28.009337 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.988467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36978 (local address 127.24.167.60:35829)
0808 12:57:27.988958 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.988966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.992346 (+ 3380us) server_negotiation.cc:407] Connection header received
0808 12:57:27.992623 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.992630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.992715 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.992792 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:27.995189 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.996097 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.998182 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.998700 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.998847 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:28.002737 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:28.002743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:28.002749 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:28.002795 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:28.006168 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:28.006989 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:28.006994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:28.007001 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:28.007183 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:28.007666 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:28.007671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:28.007675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:28.007984 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:28.008270 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:28.008682 (+ 412us) server_negotiation.cc:299] Negotiation successful
0808 12:57:28.008953 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
I20260808 12:57:28.012509 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:27.988190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.24.167.61:37779)
0808 12:57:27.988756 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:27.988766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:27.992185 (+ 3419us) server_negotiation.cc:407] Connection header received
0808 12:57:27.992549 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:27.992559 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:27.992694 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:27.992789 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:27.995824 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.996713 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.998428 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:27.999132 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:27.999277 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:28.003697 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:28.003705 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:28.003710 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:28.003756 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:28.007508 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:28.009593 (+ 2085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:28.009599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:28.009602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:28.009774 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:28.010948 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:28.010952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:28.010955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:28.011249 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:28.011492 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:28.011915 (+ 423us) server_negotiation.cc:299] Negotiation successful
0808 12:57:28.012127 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358}
I20260808 12:57:28.697170 3743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:28.696234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.2:53045)
0808 12:57:28.696796 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:28.696967 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":284,"threads_started":1}
W20260808 12:57:28.697640 3409 heartbeater.cc:644] Failed to heartbeat to 127.24.167.62:34517 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
I20260808 12:57:28.704770 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:28.703894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.3:55185)
0808 12:57:28.704398 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:28.704547 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":239,"threads_started":1}
W20260808 12:57:28.705207 3622 heartbeater.cc:644] Failed to heartbeat to 127.24.167.62:34517 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
I20260808 12:57:28.706049 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:28.705186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.1:33049)
0808 12:57:28.705665 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:28.705819 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":263,"threads_started":1}
W20260808 12:57:28.706511 3199 heartbeater.cc:644] Failed to heartbeat to 127.24.167.62:34517 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
I20260808 12:57:29.221470 3747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2cb52625c1c4008a860a8dc90be27e5)
I20260808 12:57:29.221537 3748 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2cb52625c1c4008a860a8dc90be27e5)
I20260808 12:57:29.221721 3747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } }
I20260808 12:57:29.221807 3748 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } }
I20260808 12:57:29.223021 3747 leader_election.cc:288] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2cb52625c1c4008a860a8dc90be27e5 (127.24.167.62:34517), e2bfd2b497ef40f9a6a450d9f11467fc (127.24.167.60:35829)
I20260808 12:57:29.224407 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:29.222927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36544)
0808 12:57:29.223754 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:29.224060 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":668,"threads_started":1}
I20260808 12:57:29.225592 3754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:29.222907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36558)
0808 12:57:29.225070 (+ 2163us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:29.225278 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2003,"threads_started":1}
I20260808 12:57:29.227402 3748 leader_election.cc:288] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2cb52625c1c4008a860a8dc90be27e5 (127.24.167.62:34517), 09d348e4a51a4834b6289b59a3af8e01 (127.24.167.61:37779)
W20260808 12:57:29.227349 2872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
I20260808 12:57:29.235486 3750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:29.234487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36564)
0808 12:57:29.234719 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:29.235125 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 12:57:29.235841 2872 leader_election.cc:334] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e2cb52625c1c4008a860a8dc90be27e5 (127.24.167.62:34517): Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
W20260808 12:57:29.236660 2581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
I20260808 12:57:29.239542 3749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:29.238899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36580)
0808 12:57:29.239082 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:29.239271 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 12:57:29.239827 2581 leader_election.cc:334] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e2cb52625c1c4008a860a8dc90be27e5 (127.24.167.62:34517): Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
I20260808 12:57:29.240593 3754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:29.222929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35829 (local address 127.0.0.1:36990)
0808 12:57:29.226470 (+ 3541us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:29.226502 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:29.226637 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:29.232909 (+ 6272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:29.232918 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:29.233045 (+ 127us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:29.233357 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:29.233374 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:29.235335 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:29.235341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:29.237601 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:29.237622 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:29.239400 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:29.239406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:29.239757 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:29.239800 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 12:57:29.239902 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 12:57:29.240244 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3398,"mutex_wait_us":125,"spinlock_wait_cycles":7424,"thread_start_us":1691,"threads_started":1}
I20260808 12:57:29.240667 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:29.223626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36990 (local address 127.24.167.60:35829)
0808 12:57:29.232482 (+ 8856us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:29.232488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:29.232508 (+ 20us) server_negotiation.cc:407] Connection header received
0808 12:57:29.232592 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:29.232600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:29.232698 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:29.233410 (+ 712us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:29.233715 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:29.235163 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:29.237785 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:29.239057 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:29.239193 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:29.239305 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:29.240164 (+ 859us) server_negotiation.cc:299] Negotiation successful
0808 12:57:29.240294 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8699,"thread_start_us":8547,"threads_started":1}
I20260808 12:57:29.241269 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:29.227199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.24.167.61:37779)
0808 12:57:29.232773 (+ 5574us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:29.232783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:29.232809 (+ 26us) server_negotiation.cc:407] Connection header received
0808 12:57:29.232899 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:29.232907 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:29.233547 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:29.234292 (+ 745us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:29.234567 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:29.235950 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:29.238230 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:29.239537 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:29.239674 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:29.240137 (+ 463us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:29.240741 (+ 604us) server_negotiation.cc:299] Negotiation successful
0808 12:57:29.240875 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":312,"server-negotiator.queue_time_us":5423,"spinlock_wait_cycles":1176576,"thread_start_us":5221,"threads_started":1}
I20260808 12:57:29.241565 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:29.223030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:37779 (local address 127.0.0.1:59136)
0808 12:57:29.225944 (+ 2914us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:29.225965 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:29.226143 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:29.233713 (+ 7570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:29.233718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:29.233748 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:29.234144 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:29.234158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:29.236213 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:29.236219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:29.238070 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:29.238082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:29.240145 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:29.240151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:29.240315 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:29.240355 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 12:57:29.240463 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 12:57:29.240622 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2784,"mutex_wait_us":1,"thread_start_us":706,"threads_started":1}
I20260808 12:57:29.241469 2916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09d348e4a51a4834b6289b59a3af8e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" is_pre_election: true
I20260808 12:57:29.241881 2916 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09d348e4a51a4834b6289b59a3af8e01 in term 1.
I20260808 12:57:29.242192 2619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "09d348e4a51a4834b6289b59a3af8e01" is_pre_election: true
I20260808 12:57:29.242503 2619 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2bfd2b497ef40f9a6a450d9f11467fc in term 1.
I20260808 12:57:29.242453 2585 leader_election.cc:302] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09d348e4a51a4834b6289b59a3af8e01, e2bfd2b497ef40f9a6a450d9f11467fc; no voters: e2cb52625c1c4008a860a8dc90be27e5
I20260808 12:57:29.242810 3747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 12:57:29.242959 3747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e2cb52625c1c4008a860a8dc90be27e5)
I20260808 12:57:29.243080 3747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 1 FOLLOWER]: Advancing to term 2
I20260808 12:57:29.243001 2876 leader_election.cc:302] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09d348e4a51a4834b6289b59a3af8e01, e2bfd2b497ef40f9a6a450d9f11467fc; no voters: e2cb52625c1c4008a860a8dc90be27e5
I20260808 12:57:29.243317 3748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 12:57:29.243456 3748 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 1 FOLLOWER]: Starting leader election (detected failure of leader e2cb52625c1c4008a860a8dc90be27e5)
I20260808 12:57:29.243567 3748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 1 FOLLOWER]: Advancing to term 2
I20260808 12:57:29.245851 3748 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } }
I20260808 12:57:29.245963 3747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } }
I20260808 12:57:29.246582 3748 leader_election.cc:288] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [CANDIDATE]: Term 2 election: Requested vote from peers e2cb52625c1c4008a860a8dc90be27e5 (127.24.167.62:34517), 09d348e4a51a4834b6289b59a3af8e01 (127.24.167.61:37779)
I20260808 12:57:29.246747 3747 leader_election.cc:288] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [CANDIDATE]: Term 2 election: Requested vote from peers e2cb52625c1c4008a860a8dc90be27e5 (127.24.167.62:34517), e2bfd2b497ef40f9a6a450d9f11467fc (127.24.167.60:35829)
I20260808 12:57:29.247071 2619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "09d348e4a51a4834b6289b59a3af8e01"
I20260808 12:57:29.247153 2916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09d348e4a51a4834b6289b59a3af8e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc"
I20260808 12:57:29.247334 2619 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e2bfd2b497ef40f9a6a450d9f11467fc in current term 2: Already voted for candidate 09d348e4a51a4834b6289b59a3af8e01 in this term.
I20260808 12:57:29.247416 2916 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 09d348e4a51a4834b6289b59a3af8e01 in current term 2: Already voted for candidate e2bfd2b497ef40f9a6a450d9f11467fc in this term.
I20260808 12:57:29.247534 3754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:29.247084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36594)
0808 12:57:29.247250 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:29.247349 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 12:57:29.248049 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:29.246664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36582)
0808 12:57:29.247810 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:29.247897 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1038}
I20260808 12:57:29.248673 3754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:29.248306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36602)
0808 12:57:29.248443 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:29.248525 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 12:57:29.248970 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:29.248582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36614)
0808 12:57:29.248758 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:29.248851 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260808 12:57:29.248941 2581 leader_election.cc:334] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e2cb52625c1c4008a860a8dc90be27e5 (127.24.167.62:34517): Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
I20260808 12:57:29.249073 2581 leader_election.cc:302] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09d348e4a51a4834b6289b59a3af8e01; no voters: e2bfd2b497ef40f9a6a450d9f11467fc, e2cb52625c1c4008a860a8dc90be27e5
W20260808 12:57:29.249145 2872 leader_election.cc:334] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e2cb52625c1c4008a860a8dc90be27e5 (127.24.167.62:34517): Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
I20260808 12:57:29.249274 2872 leader_election.cc:302] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2bfd2b497ef40f9a6a450d9f11467fc; no voters: 09d348e4a51a4834b6289b59a3af8e01, e2cb52625c1c4008a860a8dc90be27e5
I20260808 12:57:29.249344 3747 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 12:57:29.249584 3748 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 12:57:29.699476 3762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:29.698615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.2:55339)
0808 12:57:29.699164 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:29.699257 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":294,"threads_started":1}
I20260808 12:57:29.706828 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:29.705982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.3:33887)
0808 12:57:29.706508 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:29.706629 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":287,"threads_started":1}
I20260808 12:57:29.707913 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:29.707114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.1:50969)
0808 12:57:29.707577 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:29.707658 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
I20260808 12:57:30.701525 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:30.700623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.2:40023)
0808 12:57:30.701183 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:30.701298 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":285,"threads_started":1}
I20260808 12:57:30.709146 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:30.707997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.3:33699)
0808 12:57:30.708735 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:30.708873 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":516,"threads_started":1}
I20260808 12:57:30.710354 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:30.708920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.1:39807)
0808 12:57:30.710097 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:30.710179 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1085,"thread_start_us":989,"threads_started":1}
I20260808 12:57:30.895223 3769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:57:30.895514 3769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } }
I20260808 12:57:30.896220 3769 leader_election.cc:288] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2cb52625c1c4008a860a8dc90be27e5 (127.24.167.62:34517), e2bfd2b497ef40f9a6a450d9f11467fc (127.24.167.60:35829)
I20260808 12:57:30.896803 2916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09d348e4a51a4834b6289b59a3af8e01" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" is_pre_election: true
I20260808 12:57:30.897082 2916 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09d348e4a51a4834b6289b59a3af8e01 in term 2.
I20260808 12:57:30.897284 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:30.896358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36876)
0808 12:57:30.896896 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:30.897058 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"threads_started":1}
I20260808 12:57:30.897615 2585 leader_election.cc:302] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d348e4a51a4834b6289b59a3af8e01, e2bfd2b497ef40f9a6a450d9f11467fc; no voters:
I20260808 12:57:30.897980 3769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 12:57:30.898206 3769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:57:30.898322 3769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 2 FOLLOWER]: Advancing to term 3
I20260808 12:57:30.899184 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:30.898791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36882)
0808 12:57:30.898959 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:30.899039 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260808 12:57:30.899452 2581 leader_election.cc:334] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e2cb52625c1c4008a860a8dc90be27e5 (127.24.167.62:34517): Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
I20260808 12:57:30.900873 3769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } }
I20260808 12:57:30.901561 3769 leader_election.cc:288] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [CANDIDATE]: Term 3 election: Requested vote from peers e2cb52625c1c4008a860a8dc90be27e5 (127.24.167.62:34517), e2bfd2b497ef40f9a6a450d9f11467fc (127.24.167.60:35829)
I20260808 12:57:30.902048 2916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09d348e4a51a4834b6289b59a3af8e01" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc"
I20260808 12:57:30.902134 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:30.901701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36884)
0808 12:57:30.901892 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:30.901989 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 12:57:30.902282 2916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 2 FOLLOWER]: Advancing to term 3
I20260808 12:57:30.903159 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:30.902841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36898)
0808 12:57:30.902975 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:30.903049 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 12:57:30.903331 2581 leader_election.cc:334] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e2cb52625c1c4008a860a8dc90be27e5 (127.24.167.62:34517): Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
I20260808 12:57:30.904726 2916 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09d348e4a51a4834b6289b59a3af8e01 in term 3.
I20260808 12:57:30.905236 2585 leader_election.cc:302] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09d348e4a51a4834b6289b59a3af8e01, e2bfd2b497ef40f9a6a450d9f11467fc; no voters: e2cb52625c1c4008a860a8dc90be27e5
I20260808 12:57:30.905566 3769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 12:57:30.906011 3769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [term 3 LEADER]: Becoming Leader. State: Replica: 09d348e4a51a4834b6289b59a3af8e01, State: Running, Role: LEADER
I20260808 12:57:30.906488 3769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } }
I20260808 12:57:30.909209 3773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09d348e4a51a4834b6289b59a3af8e01. Latest consensus state: current_term: 3 leader_uuid: "09d348e4a51a4834b6289b59a3af8e01" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } } }
I20260808 12:57:30.909693 3773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:30.912043 3776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:57:30.916991 3776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:57:30.918112 3776 catalog_manager.cc:1259] Loaded cluster ID: ffe882dca60648c4b97a045851f266de
I20260808 12:57:30.918188 3776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:57:30.919553 3776 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:57:30.920457 3776 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: Loaded TSK: 0
I20260808 12:57:30.921525 3776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 12:57:30.970777 2607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:57:30.977851 2607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:57:30.982282 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:30.981843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36910)
0808 12:57:30.982032 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:30.982140 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 12:57:30.982355 2916 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [term 3 FOLLOWER]: Refusing update from remote peer 09d348e4a51a4834b6289b59a3af8e01: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260808 12:57:30.983223 3773 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 12:57:30.983474 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:30.983043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:36926)
0808 12:57:30.983228 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:30.983335 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260808 12:57:30.983625 2581 consensus_peers.cc:487] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 -> Peer e2cb52625c1c4008a860a8dc90be27e5 (127.24.167.62:34517): Couldn't send request to peer e2cb52625c1c4008a860a8dc90be27e5. Status: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 12:57:30.986635 3778 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09d348e4a51a4834b6289b59a3af8e01. Latest consensus state: current_term: 3 leader_uuid: "09d348e4a51a4834b6289b59a3af8e01" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } } }
I20260808 12:57:30.986969 3778 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [sys.catalog]: This master's current role is: FOLLOWER
I20260808 12:57:30.988261 3769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "09d348e4a51a4834b6289b59a3af8e01" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } } }
I20260808 12:57:30.988727 3769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:30.989859 3778 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "09d348e4a51a4834b6289b59a3af8e01" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2cb52625c1c4008a860a8dc90be27e5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34517 } } peers { permanent_uuid: "09d348e4a51a4834b6289b59a3af8e01" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "e2bfd2b497ef40f9a6a450d9f11467fc" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35829 } attrs { promote: false } } }
I20260808 12:57:30.990129 3778 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2bfd2b497ef40f9a6a450d9f11467fc [sys.catalog]: This master's current role is: FOLLOWER
I20260808 12:57:31.014312 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:30.996637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:32969 (local address 127.0.0.1:36790)
0808 12:57:30.997153 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:30.997183 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:30.997310 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:31.004652 (+ 7342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:31.004662 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:31.004709 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:31.005197 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:31.005215 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.007823 (+ 2608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.007834 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:31.010823 (+ 2989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:31.010849 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.013276 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.013288 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:31.013595 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:31.013657 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 12:57:31.013739 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 12:57:31.013912 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368}
I20260808 12:57:31.016548 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:30.996841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36790 (local address 127.24.167.1:32969)
0808 12:57:31.003611 (+ 6770us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:31.003622 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:31.003657 (+ 35us) server_negotiation.cc:407] Connection header received
0808 12:57:31.003771 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:31.003779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:31.003874 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:31.005446 (+ 1572us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:31.005769 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.007304 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.011158 (+ 3854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.012968 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.015586 (+ 2618us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:31.015743 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:31.015816 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 12:57:31.016051 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":560,"threads_started":1}
I20260808 12:57:31.017552 3090 tablet_service.cc:1467] Processing CreateTablet for tablet 3df76b7b94124f98b2336963026c7795 (DEFAULT_TABLE table=test-table [id=056c0ad44ead449d85c308ded51ccfa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:31.018179 3090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df76b7b94124f98b2336963026c7795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:31.022931 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:30.997560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:39209 (local address 127.0.0.1:45264)
0808 12:57:31.014676 (+ 17116us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.014704 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:31.014841 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:31.017476 (+ 2635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:31.017482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:31.017512 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:31.017686 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:31.017701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.019326 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.019335 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:31.020688 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:31.020701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.022117 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.022123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:31.022307 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:31.022333 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 12:57:31.022473 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 12:57:31.022608 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16965,"thread_start_us":9927,"threads_started":1}
I20260808 12:57:31.023725 3790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:30.997959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45264 (local address 127.24.167.3:39209)
0808 12:57:31.017180 (+ 19221us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:31.017185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:31.017202 (+ 17us) server_negotiation.cc:407] Connection header received
0808 12:57:31.017262 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:31.017266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:31.017323 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:31.017450 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:31.017888 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.019108 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.020925 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.021940 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.022092 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:31.022187 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:31.023266 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0808 12:57:31.023414 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19066,"spinlock_wait_cycles":2304,"thread_start_us":19033,"threads_started":1}
I20260808 12:57:31.024541 3526 tablet_service.cc:1467] Processing CreateTablet for tablet 3df76b7b94124f98b2336963026c7795 (DEFAULT_TABLE table=test-table [id=056c0ad44ead449d85c308ded51ccfa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:31.025072 3526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df76b7b94124f98b2336963026c7795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:31.026255 3791 tablet_bootstrap.cc:492] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9: Bootstrap starting.
I20260808 12:57:31.028223 3791 tablet_bootstrap.cc:654] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:31.030489 3791 tablet_bootstrap.cc:492] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9: No bootstrap required, opened a new log
I20260808 12:57:31.030668 3791 ts_tablet_manager.cc:1397] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260808 12:57:31.031621 3791 raft_consensus.cc:348] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } }
I20260808 12:57:31.031888 3791 raft_consensus.cc:374] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:31.031991 3791 raft_consensus.cc:729] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc076c248e2541029f9460a5295055b9, State: Initialized, Role: FOLLOWER
I20260808 12:57:31.032294 3791 consensus_queue.cc:260] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } }
I20260808 12:57:31.033355 3797 tablet_bootstrap.cc:492] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d: Bootstrap starting.
I20260808 12:57:31.036103 3797 tablet_bootstrap.cc:654] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:31.037546 3789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:30.997064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:34171 (local address 127.0.0.1:54842)
0808 12:57:31.007789 (+ 10725us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.007816 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:31.008009 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:31.031112 (+ 23103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:31.031122 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:31.031162 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:31.031723 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:31.031740 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.033673 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.033679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:31.035186 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:31.035200 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.036722 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.036729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:31.036896 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:31.036922 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 12:57:31.037037 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 12:57:31.037185 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10539,"thread_start_us":36590,"threads_started":1}
I20260808 12:57:31.037546 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:30.997515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54842 (local address 127.24.167.2:34171)
0808 12:57:31.030632 (+ 33117us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:31.030639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:31.030663 (+ 24us) server_negotiation.cc:407] Connection header received
0808 12:57:31.030744 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:31.030752 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:31.030874 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:31.031972 (+ 1098us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:31.032230 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.033511 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.035362 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.036491 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.036655 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:31.036781 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:31.037078 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 12:57:31.037218 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26512,"thread_start_us":32934,"threads_started":1}
I20260808 12:57:31.039379 3797 tablet_bootstrap.cc:492] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d: No bootstrap required, opened a new log
I20260808 12:57:31.039552 3797 ts_tablet_manager.cc:1397] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 12:57:31.039979 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.039484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.1:52381)
0808 12:57:31.039715 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.039828 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260808 12:57:31.040007 3315 tablet_service.cc:1467] Processing CreateTablet for tablet 3df76b7b94124f98b2336963026c7795 (DEFAULT_TABLE table=test-table [id=056c0ad44ead449d85c308ded51ccfa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:31.040529 3315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df76b7b94124f98b2336963026c7795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:31.040402 3797 raft_consensus.cc:348] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } }
I20260808 12:57:31.040644 3797 raft_consensus.cc:374] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:31.040750 3797 raft_consensus.cc:729] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ff2f69928b42d3b99721fa260c0c5d, State: Initialized, Role: FOLLOWER
I20260808 12:57:31.041056 3797 consensus_queue.cc:260] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } }
I20260808 12:57:31.041611 3791 ts_tablet_manager.cc:1428] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9: Time spent starting tablet: real 0.011s user 0.003s sys 0.005s
I20260808 12:57:31.042634 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.042140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.1:56085)
0808 12:57:31.042361 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.042473 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260808 12:57:31.043148 3200 heartbeater.cc:503] Master 127.24.167.61:37779 was elected leader, sending a full tablet report...
I20260808 12:57:31.043622 3797 ts_tablet_manager.cc:1428] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260808 12:57:31.045159 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.044401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.3:45413)
0808 12:57:31.044904 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.045014 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373}
I20260808 12:57:31.046630 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.046248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.3:50967)
0808 12:57:31.046416 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.046505 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 12:57:31.049762 3801 tablet_bootstrap.cc:492] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845: Bootstrap starting.
I20260808 12:57:31.051373 3801 tablet_bootstrap.cc:654] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:31.053407 3801 tablet_bootstrap.cc:492] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845: No bootstrap required, opened a new log
I20260808 12:57:31.053591 3801 ts_tablet_manager.cc:1397] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260808 12:57:31.054265 3801 raft_consensus.cc:348] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } }
I20260808 12:57:31.054553 3801 raft_consensus.cc:374] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:31.054682 3801 raft_consensus.cc:729] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927f604da71144bd8c483bee804ca845, State: Initialized, Role: FOLLOWER
I20260808 12:57:31.054973 3801 consensus_queue.cc:260] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } }
I20260808 12:57:31.055853 3801 ts_tablet_manager.cc:1428] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 12:57:31.056777 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.056201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.2:34485)
0808 12:57:31.056476 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.056596 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260808 12:57:31.058060 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.057488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.2:50341)
0808 12:57:31.057782 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.057873 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260808 12:57:31.058141 3410 heartbeater.cc:503] Master 127.24.167.61:37779 was elected leader, sending a full tablet report...
I20260808 12:57:31.213498 3798 raft_consensus.cc:482] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:57:31.213776 3798 raft_consensus.cc:504] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } }
I20260808 12:57:31.214845 3798 leader_election.cc:288] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927f604da71144bd8c483bee804ca845 (127.24.167.2:34171), 62ff2f69928b42d3b99721fa260c0c5d (127.24.167.3:39209)
I20260808 12:57:31.215520 3341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df76b7b94124f98b2336963026c7795" candidate_uuid: "bc076c248e2541029f9460a5295055b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927f604da71144bd8c483bee804ca845" is_pre_election: true
I20260808 12:57:31.215539 3550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df76b7b94124f98b2336963026c7795" candidate_uuid: "bc076c248e2541029f9460a5295055b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62ff2f69928b42d3b99721fa260c0c5d" is_pre_election: true
I20260808 12:57:31.215819 3341 raft_consensus.cc:2454] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc076c248e2541029f9460a5295055b9 in term 0.
I20260808 12:57:31.215809 3550 raft_consensus.cc:2454] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc076c248e2541029f9460a5295055b9 in term 0.
I20260808 12:57:31.216382 3015 leader_election.cc:302] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62ff2f69928b42d3b99721fa260c0c5d, bc076c248e2541029f9460a5295055b9; no voters:
I20260808 12:57:31.216739 3798 raft_consensus.cc:2781] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:57:31.216897 3798 raft_consensus.cc:482] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:57:31.217024 3798 raft_consensus.cc:3037] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:31.219776 3798 raft_consensus.cc:504] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } }
I20260808 12:57:31.220394 3798 leader_election.cc:288] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [CANDIDATE]: Term 1 election: Requested vote from peers 927f604da71144bd8c483bee804ca845 (127.24.167.2:34171), 62ff2f69928b42d3b99721fa260c0c5d (127.24.167.3:39209)
I20260808 12:57:31.221081 3341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df76b7b94124f98b2336963026c7795" candidate_uuid: "bc076c248e2541029f9460a5295055b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927f604da71144bd8c483bee804ca845"
I20260808 12:57:31.221082 3550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df76b7b94124f98b2336963026c7795" candidate_uuid: "bc076c248e2541029f9460a5295055b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62ff2f69928b42d3b99721fa260c0c5d"
I20260808 12:57:31.221333 3341 raft_consensus.cc:3037] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:31.221344 3550 raft_consensus.cc:3037] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:31.224859 3341 raft_consensus.cc:2454] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc076c248e2541029f9460a5295055b9 in term 1.
I20260808 12:57:31.225003 3550 raft_consensus.cc:2454] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc076c248e2541029f9460a5295055b9 in term 1.
I20260808 12:57:31.225567 3009 leader_election.cc:302] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 62ff2f69928b42d3b99721fa260c0c5d, 927f604da71144bd8c483bee804ca845, bc076c248e2541029f9460a5295055b9; no voters:
I20260808 12:57:31.225888 3798 raft_consensus.cc:2781] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:31.226423 3798 raft_consensus.cc:686] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [term 1 LEADER]: Becoming Leader. State: Replica: bc076c248e2541029f9460a5295055b9, State: Running, Role: LEADER
I20260808 12:57:31.227018 3798 consensus_queue.cc:237] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } } peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } }
I20260808 12:57:31.227571 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.227088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.1:48987)
0808 12:57:31.227298 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.227403 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 12:57:31.231091 2606 catalog_manager.cc:5495] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc076c248e2541029f9460a5295055b9 (127.24.167.1). New cstate: current_term: 1 leader_uuid: "bc076c248e2541029f9460a5295055b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc076c248e2541029f9460a5295055b9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 32969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 } health_report { overall_health: UNKNOWN } } }
I20260808 12:57:31.271347 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.267012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.24.167.1:32969)
0808 12:57:31.267202 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:31.267206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:31.267224 (+ 18us) server_negotiation.cc:407] Connection header received
0808 12:57:31.267392 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:31.267396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:31.267483 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:31.267559 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:57:31.267901 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.268590 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.269832 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.270278 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.270430 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:31.270780 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:57:31.270879 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:57:31.270950 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:31.271074 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 12:57:31.271139 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 12:57:31.274991 3550 raft_consensus.cc:1261] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d [term 1 FOLLOWER]: Refusing update from remote peer bc076c248e2541029f9460a5295055b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:31.275137 3341 raft_consensus.cc:1261] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845 [term 1 FOLLOWER]: Refusing update from remote peer bc076c248e2541029f9460a5295055b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:31.275833 3804 consensus_queue.cc:1035] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ff2f69928b42d3b99721fa260c0c5d" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:31.276171 3798 consensus_queue.cc:1035] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "927f604da71144bd8c483bee804ca845" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 34171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:31.278426 3623 heartbeater.cc:503] Master 127.24.167.61:37779 was elected leader, sending a full tablet report...
I20260808 12:57:31.278563 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.278101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.2:35925)
0808 12:57:31.278268 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.278362 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 12:57:31.279439 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.278988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.3:35495)
0808 12:57:31.279203 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.279300 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260808 12:57:31.283937 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.283452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.1:60643)
0808 12:57:31.283666 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.283761 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 12:57:31.284698 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.284241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.2:55287)
0808 12:57:31.284429 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.284528 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 12:57:31.286742 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.285005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.1:41707)
0808 12:57:31.286269 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.286575 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1117}
I20260808 12:57:31.287480 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.286707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.24.167.3:32869)
0808 12:57:31.287205 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.287316 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371}
I20260808 12:57:31.289799 3810 mvcc.cc:205] Tried to move back new op lower bound from 7316250014814674944 to 7316250014629433344. Current Snapshot: MvccSnapshot[applied={T|T < 7316250014814674944 or (T in {7316250014814674944})}]
I20260808 12:57:31.290743 2607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:31.291056 2607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:31.291976 3809 mvcc.cc:205] Tried to move back new op lower bound from 7316250014814674944 to 7316250014629433344. Current Snapshot: MvccSnapshot[applied={T|T < 7316250014814674944 or (T in {7316250014814674944})}]
I20260808 12:57:31.297330 2607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09d348e4a51a4834b6289b59a3af8e01: Sending DeleteTablet for 3 replicas of tablet 3df76b7b94124f98b2336963026c7795
I20260808 12:57:31.298468 3315 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df76b7b94124f98b2336963026c7795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:31 UTC) from {username='oryx', principal='oryx/127.24.167.61@KRBTEST.COM'} at 127.0.0.1:54842
I20260808 12:57:31.298527 3526 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df76b7b94124f98b2336963026c7795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:31 UTC) from {username='oryx', principal='oryx/127.24.167.61@KRBTEST.COM'} at 127.0.0.1:45264
I20260808 12:57:31.299320 3090 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df76b7b94124f98b2336963026c7795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:31 UTC) from {username='oryx', principal='oryx/127.24.167.61@KRBTEST.COM'} at 127.0.0.1:36790
I20260808 12:57:31.299577 3821 tablet_replica.cc:330] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d: stopping tablet replica
I20260808 12:57:31.299813 3821 raft_consensus.cc:2229] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:31.300143 3821 raft_consensus.cc:2258] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:31.300323 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 2855
I20260808 12:57:31.302611 3821 ts_tablet_manager.cc:1905] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:31.303740 3822 tablet_replica.cc:330] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9: stopping tablet replica
I20260808 12:57:31.304018 3822 raft_consensus.cc:2229] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:57:31.304524 3822 raft_consensus.cc:2258] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:31.307143 3820 tablet_replica.cc:330] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845: stopping tablet replica
I20260808 12:57:31.307344 3820 raft_consensus.cc:2229] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:31.307669 3820 raft_consensus.cc:2258] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:31.308451 3822 ts_tablet_manager.cc:1905] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:31.308699 3820 ts_tablet_manager.cc:1905] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:31.310564 3821 ts_tablet_manager.cc:1918] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:31.310705 3821 log.cc:1196] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/3df76b7b94124f98b2336963026c7795
I20260808 12:57:31.311194 3821 ts_tablet_manager.cc:1939] T 3df76b7b94124f98b2336963026c7795 P 62ff2f69928b42d3b99721fa260c0c5d: Deleting consensus metadata
I20260808 12:57:31.312369 2586 catalog_manager.cc:4841] TS 62ff2f69928b42d3b99721fa260c0c5d (127.24.167.3:39209): tablet 3df76b7b94124f98b2336963026c7795 (table test-table [id=056c0ad44ead449d85c308ded51ccfa1]) successfully deleted
I20260808 12:57:31.314339 3822 ts_tablet_manager.cc:1918] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:31.314525 3822 log.cc:1196] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/3df76b7b94124f98b2336963026c7795
I20260808 12:57:31.315016 3822 ts_tablet_manager.cc:1939] T 3df76b7b94124f98b2336963026c7795 P bc076c248e2541029f9460a5295055b9: Deleting consensus metadata
I20260808 12:57:31.315164 3820 ts_tablet_manager.cc:1918] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:31.315315 3820 log.cc:1196] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/3df76b7b94124f98b2336963026c7795
I20260808 12:57:31.315716 3820 ts_tablet_manager.cc:1939] T 3df76b7b94124f98b2336963026c7795 P 927f604da71144bd8c483bee804ca845: Deleting consensus metadata
I20260808 12:57:31.316203 2586 catalog_manager.cc:4841] TS bc076c248e2541029f9460a5295055b9 (127.24.167.1:32969): tablet 3df76b7b94124f98b2336963026c7795 (table test-table [id=056c0ad44ead449d85c308ded51ccfa1]) successfully deleted
I20260808 12:57:31.316780 2581 catalog_manager.cc:4841] TS 927f604da71144bd8c483bee804ca845 (127.24.167.2:34171): tablet 3df76b7b94124f98b2336963026c7795 (table test-table [id=056c0ad44ead449d85c308ded51ccfa1]) successfully deleted
I20260808 12:57:31.322183 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.301753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36810 (local address 127.24.167.1:32969)
0808 12:57:31.301930 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:31.301938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:31.303124 (+ 1186us) server_negotiation.cc:407] Connection header received
0808 12:57:31.303331 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:31.303343 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:31.303477 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:31.303609 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:31.305617 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.306343 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.311524 (+ 5181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.312257 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.312369 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:31.316302 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:31.316341 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:31.316365 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:31.316404 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:31.319354 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:31.320051 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:31.320058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:31.320061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:31.320187 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:31.320570 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:31.320582 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:31.320584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:31.320964 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:31.321296 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:31.321583 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 12:57:31.321801 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 12:57:31.322835 3192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:31.453475 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 2855
I20260808 12:57:31.484798 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 3214
I20260808 12:57:31.497896 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.485544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.24.167.2:34171)
0808 12:57:31.485810 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:31.485815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:31.485849 (+ 34us) server_negotiation.cc:407] Connection header received
0808 12:57:31.486027 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:31.486031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:31.486095 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:31.486202 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:31.487532 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.488243 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.489592 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.489977 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.490067 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:31.492663 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:31.492680 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:31.492698 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:31.492738 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:31.495478 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:31.496086 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:31.496093 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:31.496095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:31.496230 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:31.496569 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:31.496574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:31.496576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:31.496928 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:31.497188 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:31.497368 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 12:57:31.497552 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 12:57:31.498768 3402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:31.545445 3829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.544650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:52584)
0808 12:57:31.545164 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.545268 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":267,"threads_started":1}
I20260808 12:57:31.547132 3829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.546786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:34517 (local address 127.0.0.1:52596)
0808 12:57:31.546934 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:31.547020 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 12:57:31.596501 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 3214
I20260808 12:57:31.623394 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 3430
I20260808 12:57:31.637740 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:31.624205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.24.167.3:39209)
0808 12:57:31.624733 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:31.624738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:31.624753 (+ 15us) server_negotiation.cc:407] Connection header received
0808 12:57:31.624812 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:31.624815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:31.624883 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:31.625019 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:31.626551 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.627331 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.628554 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:31.628999 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:31.629166 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:31.631892 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:31.631908 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:31.631926 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:31.631973 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:31.634812 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:31.635568 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:31.635582 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:31.635585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:31.635720 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:31.636135 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:31.636140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:31.636143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:31.636536 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:31.636874 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:31.637172 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 12:57:31.637398 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":268,"threads_started":1}
I20260808 12:57:31.638495 3616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:31.737527 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 3430
I20260808 12:57:31.763311 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 2398
I20260808 12:57:31.764801 2647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:31.891871 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 2398
I20260808 12:57:31.920398 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 2548
I20260808 12:57:31.921782 2949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:32.041137 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 2548
2026-08-08T12:57:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12283 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:57:32 dist-test-slave-tk9z krb5kdc[3842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:57:32 dist-test-slave-tk9z krb5kdc[3842](info): set up 2 sockets
Aug 08 12:57:32 dist-test-slave-tk9z krb5kdc[3842](info): commencing operation
krb5kdc: starting...
W20260808 12:57:33.366830 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.007s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:57:33 dist-test-slave-tk9z krb5kdc[3842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:57:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:57:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.62.keytab.
Entry for principal oryx/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.62.keytab.
I20260808 12:57:33.633180 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:40575
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.62.keytab
--principal=oryx/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:40575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:33.981966 3863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:33.982342 3863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:33.982548 3863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:33.990097 3863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:57:33.990247 3863 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 12:57:33.990398 3863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:33.990523 3863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:33.990615 3863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:57:33.990707 3863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:57:34.001080 3863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:40575
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.62.keytab
--principal=oryx/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:40575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:34.002586 3863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:34.004027 3863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:34.016911 3873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:34.016870 3874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:34.016978 3877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:34.017840 3863 server_base.cc:1034] running on GCE node
I20260808 12:57:34.018832 3863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:34.020399 3863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:34.021682 3863 hybrid_clock.cc:648] HybridClock initialized: now 1786193854021617 us; error 41 us; skew 500 ppm
I20260808 12:57:34.022064 3863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:34 dist-test-slave-tk9z krb5kdc[3842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193854, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:34.029198 3863 init.cc:377] Logged in from keytab as oryx/127.24.167.62@KRBTEST.COM (short username oryx)
I20260808 12:57:34.030784 3863 webserver.cc:460] Webserver started at http://127.24.167.62:43661/ using document root <none> and password file <none>
I20260808 12:57:34.031488 3863 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:34.031580 3863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:34.031872 3863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:34.035008 3863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "9f4865941bc44c159ded4a459026dc23"
format_stamp: "Formatted at 2026-08-08 12:57:34 on dist-test-slave-tk9z"
server_key: "ac8c218bd83d1c803f67090a7f3ea4e3"
server_key_iv: "3ce6b1b7b80b228352117bb8b107996d"
server_key_version: "encryptionkey@0"
I20260808 12:57:34.035823 3863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "9f4865941bc44c159ded4a459026dc23"
format_stamp: "Formatted at 2026-08-08 12:57:34 on dist-test-slave-tk9z"
server_key: "ac8c218bd83d1c803f67090a7f3ea4e3"
server_key_iv: "3ce6b1b7b80b228352117bb8b107996d"
server_key_version: "encryptionkey@0"
I20260808 12:57:34.041015 3863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 12:57:34.044457 3888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:34.045868 3863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 12:57:34.046052 3863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "9f4865941bc44c159ded4a459026dc23"
format_stamp: "Formatted at 2026-08-08 12:57:34 on dist-test-slave-tk9z"
server_key: "ac8c218bd83d1c803f67090a7f3ea4e3"
server_key_iv: "3ce6b1b7b80b228352117bb8b107996d"
server_key_version: "encryptionkey@0"
I20260808 12:57:34.046309 3863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:34.065821 3863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:34.070889 3863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:34.071208 3863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:34.095317 3863 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:40575
I20260808 12:57:34.095338 3968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:40575 every 8 connection(s)
I20260808 12:57:34.096882 3863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:57:34.099562 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 3863
I20260808 12:57:34.099822 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:57:34.100540 25244 external_mini_cluster.cc:1428] Setting key 86a60ba1f21736aa154d232055148ec9
I20260808 12:57:34.101042 3970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:34.115582 3970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23: Bootstrap starting.
Aug 08 12:57:34 dist-test-slave-tk9z krb5kdc[3842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
I20260808 12:57:34.119189 3970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:34.120613 3970 log.cc:824] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:34.124154 3970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23: No bootstrap required, opened a new log
I20260808 12:57:34.126775 3975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:34.102740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.24.167.62:40575)
0808 12:57:34.103573 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:34.103588 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:34.103636 (+ 48us) server_negotiation.cc:407] Connection header received
0808 12:57:34.105975 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:34.106007 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:34.106549 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:34.107004 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:34.108416 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:34.110427 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:34.111969 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:34.113096 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:34.113291 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:34.117740 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:34.117751 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:34.117769 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:34.117824 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:34.121938 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:34.122784 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:34.122793 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:34.122801 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:34.123033 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:34.123408 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:34.123413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:34.123416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:34.124151 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:34.124441 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:34.124901 (+ 460us) server_negotiation.cc:299] Negotiation successful
0808 12:57:34.125350 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":327,"threads_started":1}
I20260808 12:57:34.132826 3970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4865941bc44c159ded4a459026dc23" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 40575 } }
I20260808 12:57:34.133212 3970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:34.133288 3970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f4865941bc44c159ded4a459026dc23, State: Initialized, Role: FOLLOWER
I20260808 12:57:34.133893 3970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4865941bc44c159ded4a459026dc23" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 40575 } }
I20260808 12:57:34.134136 3970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:57:34.134240 3970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:57:34.134361 3970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:34.137042 3970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4865941bc44c159ded4a459026dc23" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 40575 } }
I20260808 12:57:34.137532 3970 leader_election.cc:302] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f4865941bc44c159ded4a459026dc23; no voters:
I20260808 12:57:34.138104 3970 leader_election.cc:288] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:57:34.138268 3977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:34.139097 3977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [term 1 LEADER]: Becoming Leader. State: Replica: 9f4865941bc44c159ded4a459026dc23, State: Running, Role: LEADER
I20260808 12:57:34.139667 3977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4865941bc44c159ded4a459026dc23" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 40575 } }
I20260808 12:57:34.140362 3970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:57:34.142539 3980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f4865941bc44c159ded4a459026dc23. Latest consensus state: current_term: 1 leader_uuid: "9f4865941bc44c159ded4a459026dc23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4865941bc44c159ded4a459026dc23" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 40575 } } }
I20260808 12:57:34.143746 3980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:34.143335 3978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f4865941bc44c159ded4a459026dc23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4865941bc44c159ded4a459026dc23" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 40575 } } }
I20260808 12:57:34.144493 3978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23 [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:34.145846 3990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:57:34.151935 3990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:57:34.161108 3990 catalog_manager.cc:1347] Generated new cluster ID: 6c431b3a65714fb99073b2d7e741013b
I20260808 12:57:34.161262 3990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:57:34.206569 3990 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:57:34.208264 3990 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:57:34.219393 3990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23: Generated new TSK 0
I20260808 12:57:34.220583 3990 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.1.keytab.
Entry for principal oryx/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.1.keytab.
I20260808 12:57:34.291814 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:40575
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.1.keytab
--principal=oryx/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:34.639015 4009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:34.639297 4009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:34.639494 4009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:34.647430 4009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:34.647567 4009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:34.647722 4009 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 12:57:34.647852 4009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:57:34.658198 4009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.1.keytab
--principal=oryx/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.167.62:40575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:34.659596 4009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:34.660993 4009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:34.674712 4022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:34.674754 4019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:34.674721 4025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:34.675663 4009 server_base.cc:1034] running on GCE node
I20260808 12:57:34.676445 4009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:34.677428 4009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:34.678678 4009 hybrid_clock.cc:648] HybridClock initialized: now 1786193854678615 us; error 39 us; skew 500 ppm
I20260808 12:57:34.679011 4009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:34 dist-test-slave-tk9z krb5kdc[3842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193854, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:34.682184 4009 init.cc:377] Logged in from keytab as oryx/127.24.167.1@KRBTEST.COM (short username oryx)
I20260808 12:57:34.683777 4009 webserver.cc:460] Webserver started at http://127.24.167.1:35307/ using document root <none> and password file <none>
I20260808 12:57:34.684500 4009 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:34.684634 4009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:34.684973 4009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:34.688074 4009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "dcf66fd936fa42f5876a427ce101e74f"
format_stamp: "Formatted at 2026-08-08 12:57:34 on dist-test-slave-tk9z"
server_key: "e37459981831390bf18ced7f6228454a"
server_key_iv: "7e92acb8037c5427f1346a446c84332f"
server_key_version: "encryptionkey@0"
I20260808 12:57:34.688941 4009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "dcf66fd936fa42f5876a427ce101e74f"
format_stamp: "Formatted at 2026-08-08 12:57:34 on dist-test-slave-tk9z"
server_key: "e37459981831390bf18ced7f6228454a"
server_key_iv: "7e92acb8037c5427f1346a446c84332f"
server_key_version: "encryptionkey@0"
I20260808 12:57:34.694069 4009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 12:57:34.697608 4035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:34.699049 4009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:57:34.699236 4009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "dcf66fd936fa42f5876a427ce101e74f"
format_stamp: "Formatted at 2026-08-08 12:57:34 on dist-test-slave-tk9z"
server_key: "e37459981831390bf18ced7f6228454a"
server_key_iv: "7e92acb8037c5427f1346a446c84332f"
server_key_version: "encryptionkey@0"
I20260808 12:57:34.699481 4009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:34.731886 4009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:34.736124 4009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:34.736426 4009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:34.739487 4009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:34.739655 4009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 12:57:34.739868 4009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:34.739966 4009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 12:57:34 dist-test-slave-tk9z krb5kdc[3842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193854, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
I20260808 12:57:34.763193 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:34.739840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:59929 (local address 127.24.167.62:40575)
0808 12:57:34.740439 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:34.740448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:34.741726 (+ 1278us) server_negotiation.cc:407] Connection header received
0808 12:57:34.743849 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:34.743855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:34.743947 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:34.744078 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:34.746503 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:34.747705 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:34.749345 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:34.749839 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:34.750003 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:34.755218 (+ 5215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:34.755230 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:34.755239 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:34.755299 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:34.759165 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:34.760032 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:34.760040 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:34.760045 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:34.760217 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:34.760822 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:34.760830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:34.760835 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:34.761302 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:34.761482 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:34.762290 (+ 808us) server_negotiation.cc:299] Negotiation successful
0808 12:57:34.762623 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":363,"threads_started":1}
I20260808 12:57:34.763962 4051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:34.740641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:40575 (local address 127.24.167.1:59929)
0808 12:57:34.741566 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:34.741610 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:34.743565 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:34.744341 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:34.744360 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:34.745177 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:34.746163 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:34.746183 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:34.747870 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:34.747876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:34.749195 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:34.749207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:34.750149 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:34.750161 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:34.750650 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:34.751598 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:34.751651 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:34.754907 (+ 3256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:34.759326 (+ 4419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:34.759338 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:34.759360 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:34.759826 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:34.760342 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:34.760347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:34.760351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:34.760656 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:34.761473 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:34.761481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:34.761852 (+ 371us) client_negotiation.cc:769] Sending connection context
0808 12:57:34.762087 (+ 235us) client_negotiation.cc:240] Negotiation successful
0808 12:57:34.762655 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":409,"threads_started":1}
I20260808 12:57:34.788920 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:34.770346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:40575 (local address 127.24.167.1:35901)
0808 12:57:34.771445 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:34.771468 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:34.771649 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:34.772006 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:34.772017 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:34.772787 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:34.773640 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:34.773655 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:34.775239 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:34.775246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:34.777457 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:34.777484 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:34.778880 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:34.778892 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:34.779149 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:34.779958 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:34.779993 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:34.781365 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:34.785464 (+ 4099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:34.785473 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:34.785480 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:34.786209 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:34.786778 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:34.786785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:34.786792 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:34.787012 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:34.787823 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:34.787828 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:34.788040 (+ 212us) client_negotiation.cc:769] Sending connection context
0808 12:57:34.788216 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 12:57:34.788549 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":682,"threads_started":1}
I20260808 12:57:34.790148 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:34.770644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:35901 (local address 127.24.167.62:40575)
0808 12:57:34.770899 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:34.770907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:34.771572 (+ 665us) server_negotiation.cc:407] Connection header received
0808 12:57:34.771775 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:34.771778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:34.771842 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:34.771957 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:34.773846 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:34.775057 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:34.777657 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:34.778425 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:34.778583 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:34.781646 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:34.781652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:34.781656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:34.781690 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:34.785246 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:34.786435 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:34.786439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:34.786442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:34.786574 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:34.787171 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:34.787177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:34.787181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:34.787645 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:34.787846 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:34.789444 (+ 1598us) server_negotiation.cc:299] Negotiation successful
0808 12:57:34.789724 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 12:57:34.793064 4009 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:36439
I20260808 12:57:34.793087 4215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:36439 every 8 connection(s)
I20260808 12:57:34.794636 4009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:57:34.796672 4216 heartbeater.cc:347] Connected to a master server at 127.24.167.62:40575
I20260808 12:57:34.796882 4216 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:34.797562 4216 heartbeater.cc:511] Master 127.24.167.62:40575 requested a full tablet report, sending...
I20260808 12:57:34.798846 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 4009
I20260808 12:57:34.799072 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:57:34.799722 25244 external_mini_cluster.cc:1428] Setting key c95e73b2321b1321dba6c75548026f60
I20260808 12:57:34.799811 3913 ts_manager.cc:194] Registered new tserver with Master: dcf66fd936fa42f5876a427ce101e74f (127.24.167.1:36439)
I20260808 12:57:34.801527 3913 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:59929
WARNING: no policy specified for oryx/127.24.167.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.167.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.2.keytab.
Entry for principal oryx/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.2.keytab.
I20260808 12:57:34.851732 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:40575
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.2.keytab
--principal=oryx/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:35.190915 4221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:35.191241 4221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:35.191408 4221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:35.198810 4221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:35.198956 4221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:35.199095 4221 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 12:57:35.199226 4221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:57:35.208981 4221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.2.keytab
--principal=oryx/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.167.62:40575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:35.210297 4221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:35.211671 4221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:35.225939 4230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:35.226048 4227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:35.226027 4228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:35.227207 4221 server_base.cc:1034] running on GCE node
I20260808 12:57:35.227993 4221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:35.228988 4221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:35.230247 4221 hybrid_clock.cc:648] HybridClock initialized: now 1786193855230178 us; error 44 us; skew 500 ppm
I20260808 12:57:35.230616 4221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:35 dist-test-slave-tk9z krb5kdc[3842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193855, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:35.233974 4221 init.cc:377] Logged in from keytab as oryx/127.24.167.2@KRBTEST.COM (short username oryx)
I20260808 12:57:35.235544 4221 webserver.cc:460] Webserver started at http://127.24.167.2:45849/ using document root <none> and password file <none>
I20260808 12:57:35.236271 4221 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:35.236413 4221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:35.236763 4221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:35.239826 4221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "c9e48be2a62a45e2bcc413c65b39b707"
format_stamp: "Formatted at 2026-08-08 12:57:35 on dist-test-slave-tk9z"
server_key: "3ff41391d7247445413269027970927a"
server_key_iv: "a176a6e3ddf6e3c53e6fbc7e439e99d7"
server_key_version: "encryptionkey@0"
I20260808 12:57:35.240631 4221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "c9e48be2a62a45e2bcc413c65b39b707"
format_stamp: "Formatted at 2026-08-08 12:57:35 on dist-test-slave-tk9z"
server_key: "3ff41391d7247445413269027970927a"
server_key_iv: "a176a6e3ddf6e3c53e6fbc7e439e99d7"
server_key_version: "encryptionkey@0"
I20260808 12:57:35.245707 4221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 12:57:35.249308 4249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:35.250834 4221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 12:57:35.250972 4221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "c9e48be2a62a45e2bcc413c65b39b707"
format_stamp: "Formatted at 2026-08-08 12:57:35 on dist-test-slave-tk9z"
server_key: "3ff41391d7247445413269027970927a"
server_key_iv: "a176a6e3ddf6e3c53e6fbc7e439e99d7"
server_key_version: "encryptionkey@0"
I20260808 12:57:35.251164 4221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:35.268235 4221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:35.272481 4221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:35.272727 4221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:35.275535 4221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:35.275628 4221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:35.275838 4221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:35.275957 4221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 08 12:57:35 dist-test-slave-tk9z krb5kdc[3842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193855, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
I20260808 12:57:35.301954 4259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:35.276705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:40575 (local address 127.24.167.2:60447)
0808 12:57:35.277579 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:35.277628 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:35.279573 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:35.280406 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:35.280418 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:35.281206 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:35.282260 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:35.282290 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.283827 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.283832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:35.286004 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:35.286017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.286951 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.286959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:35.287196 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:35.288097 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:35.288118 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:35.291590 (+ 3472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:35.296149 (+ 4559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:35.296164 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:35.296185 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:35.297049 (+ 864us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:35.297596 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:35.297602 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:35.297605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:35.297973 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:35.298775 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:35.298786 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:35.299322 (+ 536us) client_negotiation.cc:769] Sending connection context
0808 12:57:35.299671 (+ 349us) client_negotiation.cc:240] Negotiation successful
0808 12:57:35.300243 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":375,"threads_started":1}
I20260808 12:57:35.302780 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:35.275983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:60447 (local address 127.24.167.62:40575)
0808 12:57:35.276191 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:35.276198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:35.277765 (+ 1567us) server_negotiation.cc:407] Connection header received
0808 12:57:35.279859 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:35.279868 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:35.279994 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:35.280129 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:35.282504 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.283645 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.286160 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.286675 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.286813 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:35.291833 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:35.291864 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:35.291871 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:35.291924 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:35.295903 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:35.297245 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:35.297254 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:35.297260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:35.297432 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:35.298147 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:35.298156 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:35.298161 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:35.298562 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:35.301921 (+ 3359us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:35.302124 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 12:57:35.302398 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 12:57:35.323102 4221 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:45851
I20260808 12:57:35.323159 4422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:45851 every 8 connection(s)
I20260808 12:57:35.324651 4221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:57:35.324968 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:35.308387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:47459 (local address 127.24.167.62:40575)
0808 12:57:35.308555 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:35.308559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:35.309330 (+ 771us) server_negotiation.cc:407] Connection header received
0808 12:57:35.309531 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:35.309539 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:35.309631 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:35.309743 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:35.311529 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.312551 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.314923 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.315692 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.315836 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:35.319075 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:35.319081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:35.319085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:35.319116 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:35.321970 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:35.323032 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:35.323038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:35.323041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:35.323219 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:35.323646 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:35.323649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:35.323651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:35.323980 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:35.324095 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:35.324571 (+ 476us) server_negotiation.cc:299] Negotiation successful
0808 12:57:35.324731 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 12:57:35.325438 4375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:35.308510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:40575 (local address 127.24.167.2:47459)
0808 12:57:35.309224 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:35.309239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:35.309396 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:35.309806 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:35.309813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:35.310662 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:35.311334 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:35.311357 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.312655 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.312660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:35.314694 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:35.314720 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.316163 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.316169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:35.316430 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:35.317222 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:35.317251 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:35.318763 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:35.322110 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:35.322119 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:35.322126 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:35.322854 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:35.323375 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:35.323381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:35.323384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:35.323533 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:35.324107 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:35.324114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:35.324338 (+ 224us) client_negotiation.cc:769] Sending connection context
0808 12:57:35.324479 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 12:57:35.324823 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":439,"threads_started":1}
I20260808 12:57:35.326639 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 4221
I20260808 12:57:35.326839 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:57:35.327291 25244 external_mini_cluster.cc:1428] Setting key 15de39bbfd0e5e6f6b184328535ab850
I20260808 12:57:35.328719 4423 heartbeater.cc:347] Connected to a master server at 127.24.167.62:40575
I20260808 12:57:35.328954 4423 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:35.330761 4423 heartbeater.cc:511] Master 127.24.167.62:40575 requested a full tablet report, sending...
I20260808 12:57:35.332077 3913 ts_manager.cc:194] Registered new tserver with Master: c9e48be2a62a45e2bcc413c65b39b707 (127.24.167.2:45851)
I20260808 12:57:35.332864 3913 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:60447
WARNING: no policy specified for oryx/127.24.167.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.167.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.3.keytab.
Entry for principal oryx/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.3.keytab.
I20260808 12:57:35.380824 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:40575
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.3.keytab
--principal=oryx/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:35.721858 4427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:35.722141 4427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:35.722236 4427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:35.729555 4427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:35.729674 4427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:35.729770 4427 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 12:57:35.729854 4427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:57:35.741351 4427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.3.keytab
--principal=oryx/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.167.62:40575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:35.742821 4427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:35.744323 4427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:35.759734 4437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:35.759703 4441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:35.759776 4438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:35.760804 4427 server_base.cc:1034] running on GCE node
I20260808 12:57:35.761457 4427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:35.762455 4427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:35.763741 4427 hybrid_clock.cc:648] HybridClock initialized: now 1786193855763660 us; error 50 us; skew 500 ppm
I20260808 12:57:35.764174 4427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:35 dist-test-slave-tk9z krb5kdc[3842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193855, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:35.768227 4427 init.cc:377] Logged in from keytab as oryx/127.24.167.3@KRBTEST.COM (short username oryx)
I20260808 12:57:35.769845 4427 webserver.cc:460] Webserver started at http://127.24.167.3:41575/ using document root <none> and password file <none>
I20260808 12:57:35.770700 4427 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:35.770879 4427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:35.771245 4427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:35.774641 4427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "b65c90a0e0b049e488ed71d9f36a63ef"
format_stamp: "Formatted at 2026-08-08 12:57:35 on dist-test-slave-tk9z"
server_key: "bad8643601797612fdc01f1a3d340158"
server_key_iv: "b06806eaf9cd2c289c5fc126bf435cf1"
server_key_version: "encryptionkey@0"
I20260808 12:57:35.775655 4427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "b65c90a0e0b049e488ed71d9f36a63ef"
format_stamp: "Formatted at 2026-08-08 12:57:35 on dist-test-slave-tk9z"
server_key: "bad8643601797612fdc01f1a3d340158"
server_key_iv: "b06806eaf9cd2c289c5fc126bf435cf1"
server_key_version: "encryptionkey@0"
I20260808 12:57:35.781648 4427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 12:57:35.785384 4451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:35.786770 4427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 12:57:35.786947 4427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "b65c90a0e0b049e488ed71d9f36a63ef"
format_stamp: "Formatted at 2026-08-08 12:57:35 on dist-test-slave-tk9z"
server_key: "bad8643601797612fdc01f1a3d340158"
server_key_iv: "b06806eaf9cd2c289c5fc126bf435cf1"
server_key_version: "encryptionkey@0"
I20260808 12:57:35.787199 4427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:35.804852 4216 heartbeater.cc:503] Master 127.24.167.62:40575 was elected leader, sending a full tablet report...
I20260808 12:57:35.832017 4427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:35.836584 4427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:35.836908 4427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:35.839818 4427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:35.839937 4427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:35.840178 4427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:35.840322 4427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 08 12:57:35 dist-test-slave-tk9z krb5kdc[3842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193855, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
I20260808 12:57:35.867108 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:35.840368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:43637 (local address 127.24.167.62:40575)
0808 12:57:35.841489 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:35.841494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:35.842129 (+ 635us) server_negotiation.cc:407] Connection header received
0808 12:57:35.844282 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:35.844288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:35.844381 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:35.844512 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:35.846685 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.847787 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.851291 (+ 3504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.852073 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.852222 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:35.858065 (+ 5843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:35.858071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:35.858075 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:35.858117 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:35.861996 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:35.863774 (+ 1778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:35.863778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:35.863781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:35.863935 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:35.864636 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:35.864646 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:35.864649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:35.865109 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:35.865309 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:35.866360 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0808 12:57:35.866648 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":929,"thread_start_us":803,"threads_started":1}
I20260808 12:57:35.868258 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:35.841068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:40575 (local address 127.24.167.3:43637)
0808 12:57:35.841967 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:35.842004 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:35.843991 (+ 1987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:35.844745 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:35.844753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:35.845482 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:35.846473 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:35.846497 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.847959 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.847966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:35.851087 (+ 3121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:35.851115 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.852317 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.852323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:35.852734 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:35.853725 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:35.853758 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:35.857778 (+ 4020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:35.862160 (+ 4382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:35.862173 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:35.862192 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:35.863604 (+ 1412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:35.864078 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:35.864083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:35.864087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:35.864499 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:35.865271 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:35.865294 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:35.865908 (+ 614us) client_negotiation.cc:769] Sending connection context
0808 12:57:35.866213 (+ 305us) client_negotiation.cc:240] Negotiation successful
0808 12:57:35.866934 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":445,"threads_started":1}
I20260808 12:57:35.889801 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:35.874961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:40575 (local address 127.24.167.3:55879)
0808 12:57:35.875621 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:35.875644 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:35.875822 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:35.876173 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:35.876177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:35.876666 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:35.876961 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:35.876971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.877991 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.877995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:35.879766 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:35.879783 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.880587 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.880594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:35.880770 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:35.881418 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:35.881437 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:35.882158 (+ 721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:35.886101 (+ 3943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:35.886110 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:35.886116 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:35.886855 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:35.887302 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:35.887308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:35.887315 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:35.887531 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:35.888289 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:35.888294 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:35.888550 (+ 256us) client_negotiation.cc:769] Sending connection context
0808 12:57:35.888748 (+ 198us) client_negotiation.cc:240] Negotiation successful
0808 12:57:35.889155 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":389,"threads_started":1}
I20260808 12:57:35.890831 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:35.874902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:55879 (local address 127.24.167.62:40575)
0808 12:57:35.875065 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:35.875073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:35.875734 (+ 661us) server_negotiation.cc:407] Connection header received
0808 12:57:35.875957 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:35.875962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:35.876037 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:35.876145 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:35.877109 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.877820 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.879921 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.880377 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.880512 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:35.882407 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:35.882417 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:35.882424 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:35.882463 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:35.885929 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:35.887021 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:35.887029 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:35.887033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:35.887178 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:35.887704 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:35.887711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:35.887716 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:35.888043 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:35.888225 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:35.890132 (+ 1907us) server_negotiation.cc:299] Negotiation successful
0808 12:57:35.890444 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 12:57:35.900002 4427 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:32899
I20260808 12:57:35.900033 4626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:32899 every 8 connection(s)
I20260808 12:57:35.901250 4427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:57:35.903714 4627 heartbeater.cc:347] Connected to a master server at 127.24.167.62:40575
I20260808 12:57:35.904062 4627 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:35.904978 4627 heartbeater.cc:511] Master 127.24.167.62:40575 requested a full tablet report, sending...
I20260808 12:57:35.906684 3913 ts_manager.cc:194] Registered new tserver with Master: b65c90a0e0b049e488ed71d9f36a63ef (127.24.167.3:32899)
I20260808 12:57:35.906822 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 4427
I20260808 12:57:35.907073 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:57:35.907541 25244 external_mini_cluster.cc:1428] Setting key 90f24e1c2b535c38d7ea3530171e2b72
I20260808 12:57:35.907703 3913 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:43637
I20260808 12:57:35.910812 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 12:57:35 dist-test-slave-tk9z krb5kdc[3842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 12:57:35 dist-test-slave-tk9z krb5kdc[3842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:35 dist-test-slave-tk9z krb5kdc[3842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:36.335752 4423 heartbeater.cc:503] Master 127.24.167.62:40575 was elected leader, sending a full tablet report...
I20260808 12:57:36.910869 4627 heartbeater.cc:503] Master 127.24.167.62:40575 was elected leader, sending a full tablet report...
W20260808 12:57:37.137221 4641 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.173s user 0.011s sys 0.007s
W20260808 12:57:37.138813 4464 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:35.958143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.24.167.62:40575)
0808 12:57:35.958407 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:35.958414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:35.958616 (+ 202us) server_negotiation.cc:407] Connection header received
0808 12:57:35.958859 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:35.958864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:35.958937 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:35.959049 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:35.960460 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.961198 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.962616 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:35.963033 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:35.963175 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.138355 (+1175180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37852: BlockingRecv error: recv got EOF from 127.0.0.1:37852 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 08 12:57:37 dist-test-slave-tk9z krb5kdc[3842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
I20260808 12:57:37.163249 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.143368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.24.167.62:40575)
0808 12:57:37.143648 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:37.143655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:37.143891 (+ 236us) server_negotiation.cc:407] Connection header received
0808 12:57:37.144022 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:37.144026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:37.144101 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:37.144195 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:37.145617 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.146488 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.147898 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.148356 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.148463 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.153144 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:37.153151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:37.153157 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:37.153210 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:37.159435 (+ 6225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:37.160608 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:37.160618 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:37.160625 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:37.160826 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:37.161433 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:37.161441 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:37.161455 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:37.161766 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:37.161948 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:37.162518 (+ 570us) server_negotiation.cc:299] Negotiation successful
0808 12:57:37.162768 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260808 12:57:37.167917 3913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:57:37.175232 3913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:57:37.210850 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.187673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43180 (local address 127.24.167.3:32899)
0808 12:57:37.198041 (+ 10368us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:37.198055 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:37.201073 (+ 3018us) server_negotiation.cc:407] Connection header received
0808 12:57:37.201433 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:37.201443 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:37.201705 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:37.201911 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:37.203966 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.205563 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.208119 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.209603 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.209750 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.209983 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:37.210326 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 12:57:37.210524 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10148,"thread_start_us":9999,"threads_started":1}
I20260808 12:57:37.210959 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.187429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:32899 (local address 127.0.0.1:43180)
0808 12:57:37.200871 (+ 13442us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:37.200951 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:37.201200 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:37.203226 (+ 2026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:37.203231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:37.203257 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:37.203730 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.203745 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.205691 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.205698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.207933 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.207951 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.209814 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.209821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.210025 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.210079 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 12:57:37.210209 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 12:57:37.210455 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13078,"mutex_wait_us":3,"thread_start_us":15569,"threads_started":1}
I20260808 12:57:37.211722 4658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.189648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:45851 (local address 127.0.0.1:38552)
0808 12:57:37.203251 (+ 13603us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:37.203268 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:37.203399 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:37.204059 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:37.204069 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:37.204095 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:37.204642 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.204652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.206709 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.206713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.208590 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.208605 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.210828 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.210837 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.211049 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.211093 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 12:57:37.211178 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 12:57:37.211303 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13491,"mutex_wait_us":49,"thread_start_us":10675,"threads_started":1}
I20260808 12:57:37.212047 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.189959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38552 (local address 127.24.167.2:45851)
0808 12:57:37.202819 (+ 12860us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:37.202829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:37.203428 (+ 599us) server_negotiation.cc:407] Connection header received
0808 12:57:37.203539 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:37.203547 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:37.203770 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:37.203962 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:37.204829 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.206502 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.208795 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.210228 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.210399 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.210637 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:37.211535 (+ 898us) server_negotiation.cc:299] Negotiation successful
0808 12:57:37.211731 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12690,"thread_start_us":12532,"threads_started":1}
I20260808 12:57:37.212152 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.188291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.24.167.1:36439)
0808 12:57:37.203226 (+ 14935us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:37.203235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:37.203265 (+ 30us) server_negotiation.cc:407] Connection header received
0808 12:57:37.203378 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:37.203385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:37.203601 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:37.203798 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:37.204829 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.206502 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.208942 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.210491 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.211440 (+ 949us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.211646 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:37.211722 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 12:57:37.211872 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14728,"thread_start_us":14564,"threads_started":1}
I20260808 12:57:37.212510 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.187640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:36439 (local address 127.0.0.1:60216)
0808 12:57:37.201873 (+ 14233us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:37.201895 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:37.202033 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:37.203793 (+ 1760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:37.203797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:37.203824 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:37.204622 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.204641 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.206646 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.206652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.208750 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.208766 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.210747 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.210754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.210965 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.211007 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 12:57:37.211076 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 12:57:37.211205 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14083,"mutex_wait_us":283,"thread_start_us":14489,"threads_started":1}
I20260808 12:57:37.215701 4105 tablet_service.cc:1467] Processing CreateTablet for tablet 95638201aee84e679bd726a0b1f67dab (DEFAULT_TABLE table=test-table [id=148e91f8aff540089d47c385e38c933a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:37.215643 4527 tablet_service.cc:1467] Processing CreateTablet for tablet 95638201aee84e679bd726a0b1f67dab (DEFAULT_TABLE table=test-table [id=148e91f8aff540089d47c385e38c933a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:37.216056 4323 tablet_service.cc:1467] Processing CreateTablet for tablet 95638201aee84e679bd726a0b1f67dab (DEFAULT_TABLE table=test-table [id=148e91f8aff540089d47c385e38c933a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:37.217017 4105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95638201aee84e679bd726a0b1f67dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:37.217145 4527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95638201aee84e679bd726a0b1f67dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:37.217301 4323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95638201aee84e679bd726a0b1f67dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:37.228745 4671 tablet_bootstrap.cc:492] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f: Bootstrap starting.
I20260808 12:57:37.232059 4672 tablet_bootstrap.cc:492] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef: Bootstrap starting.
I20260808 12:57:37.232613 4671 tablet_bootstrap.cc:654] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:37.232896 4673 tablet_bootstrap.cc:492] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707: Bootstrap starting.
I20260808 12:57:37.233982 4671 log.cc:824] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:37.236217 4672 tablet_bootstrap.cc:654] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:37.236384 4673 tablet_bootstrap.cc:654] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:37.237473 4673 log.cc:824] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:37.237548 4672 log.cc:824] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:37.237614 4671 tablet_bootstrap.cc:492] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f: No bootstrap required, opened a new log
I20260808 12:57:37.237880 4671 ts_tablet_manager.cc:1397] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260808 12:57:37.240620 4673 tablet_bootstrap.cc:492] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707: No bootstrap required, opened a new log
I20260808 12:57:37.240646 4672 tablet_bootstrap.cc:492] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef: No bootstrap required, opened a new log
I20260808 12:57:37.240890 4673 ts_tablet_manager.cc:1397] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 12:57:37.240890 4672 ts_tablet_manager.cc:1397] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 12:57:37.248410 4672 raft_consensus.cc:348] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } }
I20260808 12:57:37.248811 4672 raft_consensus.cc:374] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:37.248714 4671 raft_consensus.cc:348] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } }
I20260808 12:57:37.248955 4672 raft_consensus.cc:729] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65c90a0e0b049e488ed71d9f36a63ef, State: Initialized, Role: FOLLOWER
I20260808 12:57:37.249121 4671 raft_consensus.cc:374] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:37.249217 4671 raft_consensus.cc:729] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf66fd936fa42f5876a427ce101e74f, State: Initialized, Role: FOLLOWER
I20260808 12:57:37.249665 4672 consensus_queue.cc:260] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } }
I20260808 12:57:37.249890 4671 consensus_queue.cc:260] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } }
I20260808 12:57:37.251989 4671 ts_tablet_manager.cc:1428] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260808 12:57:37.251771 4673 raft_consensus.cc:348] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } }
I20260808 12:57:37.252285 4673 raft_consensus.cc:374] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:37.252454 4673 raft_consensus.cc:729] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e48be2a62a45e2bcc413c65b39b707, State: Initialized, Role: FOLLOWER
I20260808 12:57:37.253711 4672 ts_tablet_manager.cc:1428] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260808 12:57:37.255291 4673 consensus_queue.cc:260] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } }
I20260808 12:57:37.257258 4673 ts_tablet_manager.cc:1428] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260808 12:57:37.298167 4217 tablet.cc:2369] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:57:37.299549 4683 raft_consensus.cc:482] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:57:37.299860 4683 raft_consensus.cc:504] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } }
I20260808 12:57:37.301589 4683 leader_election.cc:288] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65c90a0e0b049e488ed71d9f36a63ef (127.24.167.3:32899), dcf66fd936fa42f5876a427ce101e74f (127.24.167.1:36439)
I20260808 12:57:37.309743 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.302053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:47333 (local address 127.24.167.1:36439)
0808 12:57:37.302265 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:37.302273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:37.303233 (+ 960us) server_negotiation.cc:407] Connection header received
0808 12:57:37.303555 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:37.303561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:37.303642 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:37.303742 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:37.304506 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.305236 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.307999 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.308780 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.308911 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.308973 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:37.309354 (+ 381us) server_negotiation.cc:299] Negotiation successful
0808 12:57:37.309491 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 12:57:37.309856 4685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.302108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:36439 (local address 127.24.167.2:47333)
0808 12:57:37.303104 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:37.303122 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:37.303342 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:37.303770 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:37.303773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:37.303806 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:37.304291 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.304303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.305388 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.305391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.307835 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.307850 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.309005 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.309010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.309148 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.309174 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 12:57:37.309325 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 12:57:37.309461 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":87,"thread_start_us":559,"threads_started":1}
I20260808 12:57:37.310892 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.302049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:32899 (local address 127.24.167.2:55965)
0808 12:57:37.302977 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:37.302992 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:37.303117 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:37.303612 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:37.303617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:37.303642 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:37.304123 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.304134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.305222 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.305227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.307836 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.307850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.310130 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.310136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.310318 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.310353 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 12:57:37.310475 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 12:57:37.310574 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":68,"spinlock_wait_cycles":27392,"thread_start_us":521,"threads_started":1}
I20260808 12:57:37.310925 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.302144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:55965 (local address 127.24.167.3:32899)
0808 12:57:37.302368 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:37.302373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:37.303088 (+ 715us) server_negotiation.cc:407] Connection header received
0808 12:57:37.303310 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:37.303317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:37.303420 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:37.303518 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:37.304280 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.305089 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.309045 (+ 3956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.309780 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.309990 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.310075 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:37.310550 (+ 475us) server_negotiation.cc:299] Negotiation successful
0808 12:57:37.310692 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 12:57:37.311374 4141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95638201aee84e679bd726a0b1f67dab" candidate_uuid: "c9e48be2a62a45e2bcc413c65b39b707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcf66fd936fa42f5876a427ce101e74f" is_pre_election: true
I20260808 12:57:37.311973 4141 raft_consensus.cc:2454] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9e48be2a62a45e2bcc413c65b39b707 in term 0.
I20260808 12:57:37.312057 4563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95638201aee84e679bd726a0b1f67dab" candidate_uuid: "c9e48be2a62a45e2bcc413c65b39b707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" is_pre_election: true
I20260808 12:57:37.312638 4563 raft_consensus.cc:2454] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9e48be2a62a45e2bcc413c65b39b707 in term 0.
I20260808 12:57:37.312745 4254 leader_election.cc:302] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9e48be2a62a45e2bcc413c65b39b707, dcf66fd936fa42f5876a427ce101e74f; no voters:
I20260808 12:57:37.313192 4683 raft_consensus.cc:2781] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:57:37.313344 4683 raft_consensus.cc:482] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:57:37.313459 4683 raft_consensus.cc:3037] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:37.315771 4683 raft_consensus.cc:504] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } }
I20260808 12:57:37.316431 4683 leader_election.cc:288] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [CANDIDATE]: Term 1 election: Requested vote from peers b65c90a0e0b049e488ed71d9f36a63ef (127.24.167.3:32899), dcf66fd936fa42f5876a427ce101e74f (127.24.167.1:36439)
I20260808 12:57:37.316862 4563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95638201aee84e679bd726a0b1f67dab" candidate_uuid: "c9e48be2a62a45e2bcc413c65b39b707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65c90a0e0b049e488ed71d9f36a63ef"
I20260808 12:57:37.316915 4141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95638201aee84e679bd726a0b1f67dab" candidate_uuid: "c9e48be2a62a45e2bcc413c65b39b707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcf66fd936fa42f5876a427ce101e74f"
I20260808 12:57:37.317090 4563 raft_consensus.cc:3037] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:37.317129 4141 raft_consensus.cc:3037] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:37.320393 4141 raft_consensus.cc:2454] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e48be2a62a45e2bcc413c65b39b707 in term 1.
I20260808 12:57:37.320393 4563 raft_consensus.cc:2454] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e48be2a62a45e2bcc413c65b39b707 in term 1.
I20260808 12:57:37.320863 4254 leader_election.cc:302] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9e48be2a62a45e2bcc413c65b39b707, dcf66fd936fa42f5876a427ce101e74f; no voters:
I20260808 12:57:37.321144 4683 raft_consensus.cc:2781] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:37.321647 4683 raft_consensus.cc:686] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [term 1 LEADER]: Becoming Leader. State: Replica: c9e48be2a62a45e2bcc413c65b39b707, State: Running, Role: LEADER
I20260808 12:57:37.322232 4683 consensus_queue.cc:237] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } }
I20260808 12:57:37.327019 3913 catalog_manager.cc:5495] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9e48be2a62a45e2bcc413c65b39b707 (127.24.167.2). New cstate: current_term: 1 leader_uuid: "c9e48be2a62a45e2bcc413c65b39b707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } health_report { overall_health: HEALTHY } } }
W20260808 12:57:37.328665 4424 tablet.cc:2369] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:57:37.362465 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.357014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38564 (local address 127.24.167.2:45851)
0808 12:57:37.357233 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:37.357239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:37.357259 (+ 20us) server_negotiation.cc:407] Connection header received
0808 12:57:37.357522 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:37.357526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:37.357605 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:37.357766 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:57:37.358140 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.358930 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.360280 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.360698 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.360842 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.361304 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:57:37.361847 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:57:37.362034 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:37.362096 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 12:57:37.362176 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 12:57:37.370600 4141 raft_consensus.cc:1261] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Refusing update from remote peer c9e48be2a62a45e2bcc413c65b39b707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:37.370599 4563 raft_consensus.cc:1261] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Refusing update from remote peer c9e48be2a62a45e2bcc413c65b39b707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:37.371939 4687 consensus_queue.cc:1035] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:37.372332 4683 consensus_queue.cc:1035] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:37.385172 4696 mvcc.cc:205] Tried to move back new op lower bound from 7316250039779766272 to 7316250039596355584. Current Snapshot: MvccSnapshot[applied={T|T < 7316250039779766272}]
I20260808 12:57:37.398159 3912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:37.398545 3912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:37.402628 3912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23: Sending DeleteTablet for 3 replicas of tablet 95638201aee84e679bd726a0b1f67dab
I20260808 12:57:37.403936 4527 tablet_service.cc:1514] Processing DeleteTablet for tablet 95638201aee84e679bd726a0b1f67dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:37 UTC) from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:43180
I20260808 12:57:37.403962 4105 tablet_service.cc:1514] Processing DeleteTablet for tablet 95638201aee84e679bd726a0b1f67dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:37 UTC) from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:60216
I20260808 12:57:37.404143 4323 tablet_service.cc:1514] Processing DeleteTablet for tablet 95638201aee84e679bd726a0b1f67dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:37 UTC) from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:38552
I20260808 12:57:37.406791 3913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:57:37.407672 3913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:57:37.419219 4103 tablet_service.cc:1467] Processing CreateTablet for tablet f7a2f096f7f040cb951fd12627bc3473 (DEFAULT_TABLE table=test-table [id=f9c5d435e24e4f589658b8fb6aefb255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:37.419917 4712 tablet_replica.cc:330] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707: stopping tablet replica
I20260808 12:57:37.420022 4103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a2f096f7f040cb951fd12627bc3473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:37.420445 4712 raft_consensus.cc:2229] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:57:37.421243 4712 raft_consensus.cc:2258] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:37.424547 4526 tablet_service.cc:1467] Processing CreateTablet for tablet f7a2f096f7f040cb951fd12627bc3473 (DEFAULT_TABLE table=test-table [id=f9c5d435e24e4f589658b8fb6aefb255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:37.425274 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a2f096f7f040cb951fd12627bc3473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:37.430670 4323 tablet_service.cc:1467] Processing CreateTablet for tablet f7a2f096f7f040cb951fd12627bc3473 (DEFAULT_TABLE table=test-table [id=f9c5d435e24e4f589658b8fb6aefb255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:37.431154 4323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a2f096f7f040cb951fd12627bc3473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:37.439495 4708 tablet_replica.cc:330] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef: stopping tablet replica
I20260808 12:57:37.440130 4708 raft_consensus.cc:2229] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:37.441932 4707 tablet_replica.cc:330] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f: stopping tablet replica
I20260808 12:57:37.442386 4707 raft_consensus.cc:2229] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:37.442914 4707 raft_consensus.cc:2258] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:37.443374 4671 tablet_bootstrap.cc:492] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f: Bootstrap starting.
I20260808 12:57:37.443606 4708 raft_consensus.cc:2258] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:37.444375 4672 tablet_bootstrap.cc:492] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef: Bootstrap starting.
I20260808 12:57:37.445235 4671 tablet_bootstrap.cc:654] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:37.445950 4708 ts_tablet_manager.cc:1905] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:37.446043 4672 tablet_bootstrap.cc:654] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:37.450452 4707 ts_tablet_manager.cc:1905] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:37.451974 4712 ts_tablet_manager.cc:1905] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:37.458794 4673 tablet_bootstrap.cc:492] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707: Bootstrap starting.
I20260808 12:57:37.459544 4672 tablet_bootstrap.cc:492] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef: No bootstrap required, opened a new log
I20260808 12:57:37.459720 4672 ts_tablet_manager.cc:1397] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.006s
I20260808 12:57:37.460580 4673 tablet_bootstrap.cc:654] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:37.460660 4672 raft_consensus.cc:348] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.460906 4672 raft_consensus.cc:374] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:37.461016 4672 raft_consensus.cc:729] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65c90a0e0b049e488ed71d9f36a63ef, State: Initialized, Role: FOLLOWER
I20260808 12:57:37.461377 4672 consensus_queue.cc:260] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.462065 4672 ts_tablet_manager.cc:1428] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260808 12:57:37.465530 4712 ts_tablet_manager.cc:1918] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:37.465725 4712 log.cc:1196] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/95638201aee84e679bd726a0b1f67dab
I20260808 12:57:37.466284 4712 ts_tablet_manager.cc:1939] T 95638201aee84e679bd726a0b1f67dab P c9e48be2a62a45e2bcc413c65b39b707: Deleting consensus metadata
I20260808 12:57:37.468413 3891 catalog_manager.cc:4841] TS c9e48be2a62a45e2bcc413c65b39b707 (127.24.167.2:45851): tablet 95638201aee84e679bd726a0b1f67dab (table test-table [id=148e91f8aff540089d47c385e38c933a]) successfully deleted
I20260808 12:57:37.472885 4708 ts_tablet_manager.cc:1918] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:37.473099 4708 log.cc:1196] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/95638201aee84e679bd726a0b1f67dab
I20260808 12:57:37.473666 4673 tablet_bootstrap.cc:492] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707: No bootstrap required, opened a new log
I20260808 12:57:37.473773 4708 ts_tablet_manager.cc:1939] T 95638201aee84e679bd726a0b1f67dab P b65c90a0e0b049e488ed71d9f36a63ef: Deleting consensus metadata
I20260808 12:57:37.473860 4673 ts_tablet_manager.cc:1397] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260808 12:57:37.474840 4673 raft_consensus.cc:348] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.475086 4673 raft_consensus.cc:374] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:37.475185 4673 raft_consensus.cc:729] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e48be2a62a45e2bcc413c65b39b707, State: Initialized, Role: FOLLOWER
I20260808 12:57:37.475510 4673 consensus_queue.cc:260] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.475823 3890 catalog_manager.cc:4841] TS b65c90a0e0b049e488ed71d9f36a63ef (127.24.167.3:32899): tablet 95638201aee84e679bd726a0b1f67dab (table test-table [id=148e91f8aff540089d47c385e38c933a]) successfully deleted
I20260808 12:57:37.476233 4673 ts_tablet_manager.cc:1428] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 12:57:37.477581 4671 tablet_bootstrap.cc:492] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f: No bootstrap required, opened a new log
I20260808 12:57:37.477840 4671 ts_tablet_manager.cc:1397] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.003s
I20260808 12:57:37.478863 4671 raft_consensus.cc:348] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.479171 4671 raft_consensus.cc:374] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:37.479328 4671 raft_consensus.cc:729] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf66fd936fa42f5876a427ce101e74f, State: Initialized, Role: FOLLOWER
I20260808 12:57:37.479619 4707 ts_tablet_manager.cc:1918] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:37.479859 4707 log.cc:1196] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/95638201aee84e679bd726a0b1f67dab
I20260808 12:57:37.479774 4671 consensus_queue.cc:260] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.480497 4671 ts_tablet_manager.cc:1428] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 12:57:37.480628 4707 ts_tablet_manager.cc:1939] T 95638201aee84e679bd726a0b1f67dab P dcf66fd936fa42f5876a427ce101e74f: Deleting consensus metadata
I20260808 12:57:37.482517 3893 catalog_manager.cc:4841] TS dcf66fd936fa42f5876a427ce101e74f (127.24.167.1:36439): tablet 95638201aee84e679bd726a0b1f67dab (table test-table [id=148e91f8aff540089d47c385e38c933a]) successfully deleted
W20260808 12:57:37.656095 4628 tablet.cc:2369] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:57:37.743937 4680 raft_consensus.cc:482] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:57:37.744197 4680 raft_consensus.cc:504] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.745932 4680 leader_election.cc:288] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcf66fd936fa42f5876a427ce101e74f (127.24.167.1:36439), c9e48be2a62a45e2bcc413c65b39b707 (127.24.167.2:45851)
I20260808 12:57:37.754896 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.746891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:57361 (local address 127.24.167.1:36439)
0808 12:57:37.747079 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:37.747085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:37.747683 (+ 598us) server_negotiation.cc:407] Connection header received
0808 12:57:37.747931 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:37.747938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:37.748014 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:37.748106 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:37.748925 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.750138 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.752430 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.753642 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.753828 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.753910 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:37.754340 (+ 430us) server_negotiation.cc:299] Negotiation successful
0808 12:57:37.754516 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 12:57:37.755163 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.746439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:37169 (local address 127.24.167.2:45851)
0808 12:57:37.746676 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:37.746684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:37.747334 (+ 650us) server_negotiation.cc:407] Connection header received
0808 12:57:37.747666 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:37.747673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:37.747820 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:37.747939 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:37.748654 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.749891 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.752761 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.754084 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.754281 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.754363 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:37.754709 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 12:57:37.754839 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 12:57:37.755234 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.746222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:45851 (local address 127.24.167.3:37169)
0808 12:57:37.747200 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:37.747225 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:37.747481 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:37.747974 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:37.747979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:37.748004 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:37.748458 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.748470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.750053 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.750058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.752569 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.752585 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.754351 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.754357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.754541 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.754565 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 12:57:37.754675 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 12:57:37.754810 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":37,"thread_start_us":658,"threads_started":1}
I20260808 12:57:37.756251 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.746288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:36439 (local address 127.24.167.3:57361)
0808 12:57:37.747564 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:37.747582 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:37.747744 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:37.748156 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:37.748161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:37.748195 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:37.748729 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.748743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.750285 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.750292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.752271 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.752285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.753901 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.753909 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.754112 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.754150 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 12:57:37.755718 (+ 1568us) client_negotiation.cc:240] Negotiation successful
0808 12:57:37.755871 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"mutex_wait_us":24,"thread_start_us":943,"threads_started":1}
I20260808 12:57:37.756196 4351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a2f096f7f040cb951fd12627bc3473" candidate_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e48be2a62a45e2bcc413c65b39b707" is_pre_election: true
I20260808 12:57:37.756711 4351 raft_consensus.cc:2454] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65c90a0e0b049e488ed71d9f36a63ef in term 0.
I20260808 12:57:37.756973 4141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a2f096f7f040cb951fd12627bc3473" candidate_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcf66fd936fa42f5876a427ce101e74f" is_pre_election: true
I20260808 12:57:37.757241 4141 raft_consensus.cc:2454] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65c90a0e0b049e488ed71d9f36a63ef in term 0.
I20260808 12:57:37.757561 4455 leader_election.cc:302] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65c90a0e0b049e488ed71d9f36a63ef, c9e48be2a62a45e2bcc413c65b39b707; no voters:
I20260808 12:57:37.758067 4680 raft_consensus.cc:2781] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:57:37.758196 4680 raft_consensus.cc:482] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:57:37.758299 4680 raft_consensus.cc:3037] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:37.760489 4680 raft_consensus.cc:504] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.761140 4680 leader_election.cc:288] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [CANDIDATE]: Term 1 election: Requested vote from peers dcf66fd936fa42f5876a427ce101e74f (127.24.167.1:36439), c9e48be2a62a45e2bcc413c65b39b707 (127.24.167.2:45851)
I20260808 12:57:37.761526 4141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a2f096f7f040cb951fd12627bc3473" candidate_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcf66fd936fa42f5876a427ce101e74f"
I20260808 12:57:37.761742 4141 raft_consensus.cc:3037] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:37.761858 4351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a2f096f7f040cb951fd12627bc3473" candidate_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e48be2a62a45e2bcc413c65b39b707"
I20260808 12:57:37.762082 4351 raft_consensus.cc:3037] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:37.764779 4141 raft_consensus.cc:2454] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65c90a0e0b049e488ed71d9f36a63ef in term 1.
I20260808 12:57:37.765199 4459 leader_election.cc:302] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65c90a0e0b049e488ed71d9f36a63ef, dcf66fd936fa42f5876a427ce101e74f; no voters:
I20260808 12:57:37.765483 4351 raft_consensus.cc:2454] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65c90a0e0b049e488ed71d9f36a63ef in term 1.
I20260808 12:57:37.765538 4680 raft_consensus.cc:2781] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:37.766223 4680 raft_consensus.cc:686] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [term 1 LEADER]: Becoming Leader. State: Replica: b65c90a0e0b049e488ed71d9f36a63ef, State: Running, Role: LEADER
I20260808 12:57:37.767078 4680 consensus_queue.cc:237] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.771049 3912 catalog_manager.cc:5495] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef reported cstate change: term changed from 0 to 1, leader changed from <none> to b65c90a0e0b049e488ed71d9f36a63ef (127.24.167.3). New cstate: current_term: 1 leader_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } health_report { overall_health: HEALTHY } } }
Aug 08 12:57:37 dist-test-slave-tk9z krb5kdc[3842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193854, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.62@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
I20260808 12:57:37.841158 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.825172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:40575 (local address 127.0.0.1:37862)
0808 12:57:37.825930 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:37.825952 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:37.826120 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:37.826626 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:37.826631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:37.827571 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:37.827831 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.827842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.828982 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.828987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.830567 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:37.830582 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.831317 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.831323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:37.831539 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.832232 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:37.832257 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:37.835081 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:37.838107 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:37.838121 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:37.838129 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:37.838741 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:37.839222 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:37.839227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:37.839230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:37.839401 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:37.839971 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:37.839979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:37.840205 (+ 226us) client_negotiation.cc:769] Sending connection context
0808 12:57:37.840376 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 12:57:37.840681 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":392,"threads_started":1}
I20260808 12:57:37.841455 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:37.825208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.24.167.62:40575)
0808 12:57:37.826278 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:37.826282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:37.826303 (+ 21us) server_negotiation.cc:407] Connection header received
0808 12:57:37.826349 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:37.826352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:37.826466 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:37.826557 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:37.828067 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.828853 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.830710 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:37.831132 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:37.831259 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:37.835273 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:37.835279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:37.835282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:37.835311 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:37.837963 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:37.838874 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:37.838878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:37.838880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:37.839074 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:37.839536 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:37.839543 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:37.839546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:37.839836 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:37.840015 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:37.840487 (+ 472us) server_negotiation.cc:299] Negotiation successful
0808 12:57:37.840888 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":141,"server-negotiator.queue_time_us":976,"thread_start_us":795,"threads_started":1}
I20260808 12:57:37.847128 3912 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:37862:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 12:57:37.848394 3912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:57:37.855985 4103 tablet_service.cc:1467] Processing CreateTablet for tablet 38d9941dec1a4dfeadf1e77dc072a0aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67d726bfe59e49248d612d0ee9d7c9ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 12:57:37.856320 4526 tablet_service.cc:1467] Processing CreateTablet for tablet 38d9941dec1a4dfeadf1e77dc072a0aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67d726bfe59e49248d612d0ee9d7c9ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 12:57:37.856631 4103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d9941dec1a4dfeadf1e77dc072a0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:37.856593 4323 tablet_service.cc:1467] Processing CreateTablet for tablet 38d9941dec1a4dfeadf1e77dc072a0aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67d726bfe59e49248d612d0ee9d7c9ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 12:57:37.856918 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d9941dec1a4dfeadf1e77dc072a0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:37.857123 4323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d9941dec1a4dfeadf1e77dc072a0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:37.862960 4672 tablet_bootstrap.cc:492] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef: Bootstrap starting.
I20260808 12:57:37.864070 4673 tablet_bootstrap.cc:492] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707: Bootstrap starting.
I20260808 12:57:37.864591 4672 tablet_bootstrap.cc:654] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:37.865865 4673 tablet_bootstrap.cc:654] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:37.866691 4671 tablet_bootstrap.cc:492] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f: Bootstrap starting.
I20260808 12:57:37.868441 4671 tablet_bootstrap.cc:654] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:37.875051 4672 tablet_bootstrap.cc:492] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef: No bootstrap required, opened a new log
I20260808 12:57:37.875200 4671 tablet_bootstrap.cc:492] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f: No bootstrap required, opened a new log
I20260808 12:57:37.875349 4672 ts_tablet_manager.cc:1397] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260808 12:57:37.875379 4671 ts_tablet_manager.cc:1397] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260808 12:57:37.875970 4673 tablet_bootstrap.cc:492] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707: No bootstrap required, opened a new log
I20260808 12:57:37.875972 4672 raft_consensus.cc:348] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.876153 4673 ts_tablet_manager.cc:1397] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260808 12:57:37.876246 4672 raft_consensus.cc:374] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:37.876145 4671 raft_consensus.cc:348] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.876351 4672 raft_consensus.cc:729] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65c90a0e0b049e488ed71d9f36a63ef, State: Initialized, Role: FOLLOWER
I20260808 12:57:37.876394 4671 raft_consensus.cc:374] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:37.876477 4671 raft_consensus.cc:729] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf66fd936fa42f5876a427ce101e74f, State: Initialized, Role: FOLLOWER
I20260808 12:57:37.876626 4672 consensus_queue.cc:260] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.876995 4673 raft_consensus.cc:348] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.877228 4672 ts_tablet_manager.cc:1428] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260808 12:57:37.877077 4680 tablet_replica.cc:439] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } }
I20260808 12:57:37.877246 4673 raft_consensus.cc:374] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:37.877326 4673 raft_consensus.cc:729] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e48be2a62a45e2bcc413c65b39b707, State: Initialized, Role: FOLLOWER
I20260808 12:57:37.877386 4680 tablet_replica.cc:442] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:57:37.876789 4671 consensus_queue.cc:260] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.877580 4673 consensus_queue.cc:260] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.878084 4671 ts_tablet_manager.cc:1428] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 12:57:37.877950 4682 tablet_replica.cc:439] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } }
I20260808 12:57:37.878358 4682 tablet_replica.cc:442] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:57:37.878624 4687 tablet_replica.cc:439] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } }
I20260808 12:57:37.878931 4687 tablet_replica.cc:442] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:57:37.879371 4673 ts_tablet_manager.cc:1428] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 12:57:37.994700 4687 raft_consensus.cc:482] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:57:37.994931 4687 raft_consensus.cc:504] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:37.995852 4687 leader_election.cc:288] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcf66fd936fa42f5876a427ce101e74f (127.24.167.1:36439), b65c90a0e0b049e488ed71d9f36a63ef (127.24.167.3:32899)
I20260808 12:57:37.996426 4141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d9941dec1a4dfeadf1e77dc072a0aa" candidate_uuid: "c9e48be2a62a45e2bcc413c65b39b707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcf66fd936fa42f5876a427ce101e74f" is_pre_election: true
I20260808 12:57:37.996474 4563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d9941dec1a4dfeadf1e77dc072a0aa" candidate_uuid: "c9e48be2a62a45e2bcc413c65b39b707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" is_pre_election: true
I20260808 12:57:37.996752 4563 raft_consensus.cc:2454] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9e48be2a62a45e2bcc413c65b39b707 in term 0.
I20260808 12:57:37.996748 4141 raft_consensus.cc:2454] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9e48be2a62a45e2bcc413c65b39b707 in term 0.
I20260808 12:57:37.997251 4254 leader_election.cc:302] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9e48be2a62a45e2bcc413c65b39b707, dcf66fd936fa42f5876a427ce101e74f; no voters:
I20260808 12:57:37.997589 4687 raft_consensus.cc:2781] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:57:37.997706 4687 raft_consensus.cc:482] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:57:37.997839 4687 raft_consensus.cc:3037] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:37.999905 4687 raft_consensus.cc:504] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.000489 4687 leader_election.cc:288] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [CANDIDATE]: Term 1 election: Requested vote from peers dcf66fd936fa42f5876a427ce101e74f (127.24.167.1:36439), b65c90a0e0b049e488ed71d9f36a63ef (127.24.167.3:32899)
I20260808 12:57:38.001051 4141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d9941dec1a4dfeadf1e77dc072a0aa" candidate_uuid: "c9e48be2a62a45e2bcc413c65b39b707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcf66fd936fa42f5876a427ce101e74f"
I20260808 12:57:38.001084 4563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d9941dec1a4dfeadf1e77dc072a0aa" candidate_uuid: "c9e48be2a62a45e2bcc413c65b39b707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65c90a0e0b049e488ed71d9f36a63ef"
I20260808 12:57:38.001302 4141 raft_consensus.cc:3037] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:38.001298 4563 raft_consensus.cc:3037] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:38.003566 4141 raft_consensus.cc:2454] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e48be2a62a45e2bcc413c65b39b707 in term 1.
I20260808 12:57:38.003587 4563 raft_consensus.cc:2454] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e48be2a62a45e2bcc413c65b39b707 in term 1.
I20260808 12:57:38.004130 4250 leader_election.cc:302] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65c90a0e0b049e488ed71d9f36a63ef, c9e48be2a62a45e2bcc413c65b39b707; no voters:
I20260808 12:57:38.004453 4687 raft_consensus.cc:2781] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:38.004849 4687 raft_consensus.cc:686] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [term 1 LEADER]: Becoming Leader. State: Replica: c9e48be2a62a45e2bcc413c65b39b707, State: Running, Role: LEADER
I20260808 12:57:38.005218 4687 consensus_queue.cc:237] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.006865 4737 tablet_replica.cc:439] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707: TxnStatusTablet state changed. Reason: New leader c9e48be2a62a45e2bcc413c65b39b707. Latest consensus state: current_term: 1 leader_uuid: "c9e48be2a62a45e2bcc413c65b39b707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } }
I20260808 12:57:38.007153 4737 tablet_replica.cc:442] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707: This TxnStatusTablet replica's current role is: LEADER
I20260808 12:57:38.007961 4739 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 12:57:38.008198 4739 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 12:57:38.008455 3907 catalog_manager.cc:5495] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9e48be2a62a45e2bcc413c65b39b707 (127.24.167.2). New cstate: current_term: 1 leader_uuid: "c9e48be2a62a45e2bcc413c65b39b707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } health_report { overall_health: UNKNOWN } } }
I20260808 12:57:38.061156 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.055511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:45851 (local address 127.0.0.1:38576)
0808 12:57:38.055755 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:38.055781 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:38.055907 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:38.056313 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:38.056316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:38.056325 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 12:57:38.056491 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:38.056498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.057982 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.057986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:38.059071 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:38.059085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.060076 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.060080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:38.060244 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.060299 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 12:57:38.060727 (+ 428us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 12:57:38.060743 (+ 16us) client_negotiation.cc:769] Sending connection context
0808 12:57:38.060802 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 12:57:38.060915 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 12:57:38.061242 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.055636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.24.167.2:45851)
0808 12:57:38.055887 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:38.055893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:38.055915 (+ 22us) server_negotiation.cc:407] Connection header received
0808 12:57:38.056085 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:38.056091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:38.056182 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:38.056281 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:57:38.056633 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.057844 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.059211 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.059924 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.060044 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.060492 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:57:38.060629 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:57:38.060725 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:38.060854 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 12:57:38.060938 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 12:57:38.065495 4563 raft_consensus.cc:1261] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Refusing update from remote peer c9e48be2a62a45e2bcc413c65b39b707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:38.065999 4141 raft_consensus.cc:1261] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Refusing update from remote peer c9e48be2a62a45e2bcc413c65b39b707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:38.066258 4737 consensus_queue.cc:1035] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:38.066627 4687 consensus_queue.cc:1035] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:38.074251 4680 tablet_replica.cc:439] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef: TxnStatusTablet state changed. Reason: New leader c9e48be2a62a45e2bcc413c65b39b707. Latest consensus state: current_term: 1 leader_uuid: "c9e48be2a62a45e2bcc413c65b39b707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } }
I20260808 12:57:38.074651 4680 tablet_replica.cc:442] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:57:38.077152 4743 mvcc.cc:205] Tried to move back new op lower bound from 7316250042630012928 to 7316250042391764992. Current Snapshot: MvccSnapshot[applied={T|T < 7316250042630012928}]
I20260808 12:57:38.078413 4682 tablet_replica.cc:439] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f: TxnStatusTablet state changed. Reason: New leader c9e48be2a62a45e2bcc413c65b39b707. Latest consensus state: current_term: 1 leader_uuid: "c9e48be2a62a45e2bcc413c65b39b707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } }
I20260808 12:57:38.078713 4682 tablet_replica.cc:442] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:57:38.083739 4687 tablet_replica.cc:439] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9e48be2a62a45e2bcc413c65b39b707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } }
I20260808 12:57:38.084111 4687 tablet_replica.cc:442] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707: This TxnStatusTablet replica's current role is: LEADER
I20260808 12:57:38.090909 4680 tablet_replica.cc:439] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9e48be2a62a45e2bcc413c65b39b707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } }
I20260808 12:57:38.091228 4680 tablet_replica.cc:442] T 38d9941dec1a4dfeadf1e77dc072a0aa P b65c90a0e0b049e488ed71d9f36a63ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:57:38.091339 4687 tablet_replica.cc:439] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9e48be2a62a45e2bcc413c65b39b707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } }
I20260808 12:57:38.091745 4687 tablet_replica.cc:442] T 38d9941dec1a4dfeadf1e77dc072a0aa P c9e48be2a62a45e2bcc413c65b39b707: This TxnStatusTablet replica's current role is: LEADER
I20260808 12:57:38.095573 4682 tablet_replica.cc:439] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9e48be2a62a45e2bcc413c65b39b707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } } }
I20260808 12:57:38.095852 4682 tablet_replica.cc:442] T 38d9941dec1a4dfeadf1e77dc072a0aa P dcf66fd936fa42f5876a427ce101e74f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 12:57:38.098636 4742 mvcc.cc:205] Tried to move back new op lower bound from 7316250042630012928 to 7316250042391764992. Current Snapshot: MvccSnapshot[applied={T|T < 7316250042630012928 or (T in {7316250042630012928})}]
I20260808 12:57:38.120081 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.100096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.24.167.3:32899)
0808 12:57:38.113683 (+ 13587us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:38.113693 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:38.114439 (+ 746us) server_negotiation.cc:407] Connection header received
0808 12:57:38.114725 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:38.114732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:38.114840 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:38.115035 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:57:38.115549 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.116327 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.117670 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.118115 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.118253 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.118776 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:57:38.119365 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:57:38.119539 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:38.119601 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 12:57:38.119690 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13329,"thread_start_us":14537,"threads_started":1}
I20260808 12:57:38.137493 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.131197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:58115 (local address 127.24.167.2:45851)
0808 12:57:38.131409 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:38.131417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:38.131736 (+ 319us) server_negotiation.cc:407] Connection header received
0808 12:57:38.131884 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:38.131891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:38.131993 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:38.132093 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:57:38.132555 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.133718 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.135287 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.135994 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.136136 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.136607 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:57:38.136766 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:57:38.136861 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:38.137042 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 12:57:38.137133 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 12:57:38.137645 4759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.131079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:45851 (local address 127.24.167.3:58115)
0808 12:57:38.131631 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:38.131647 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:38.131772 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:38.132114 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:38.132118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:38.132127 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 12:57:38.132410 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:38.132417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.133893 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.133898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:38.135139 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:38.135155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.136201 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.136206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:38.136346 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.136415 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 12:57:38.136899 (+ 484us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 12:57:38.136910 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 12:57:38.136986 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 12:57:38.137308 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":355,"threads_started":1}
I20260808 12:57:38.148253 4141 raft_consensus.cc:1261] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Refusing update from remote peer b65c90a0e0b049e488ed71d9f36a63ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:38.148633 4351 raft_consensus.cc:1261] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Refusing update from remote peer b65c90a0e0b049e488ed71d9f36a63ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:38.149250 4720 consensus_queue.cc:1035] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:38.149748 4680 consensus_queue.cc:1035] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:38.188921 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.181827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:54413 (local address 127.24.167.3:32899)
0808 12:57:38.181994 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:38.182000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:38.182687 (+ 687us) server_negotiation.cc:407] Connection header received
0808 12:57:38.182890 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:38.182896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:38.182988 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:38.183098 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:57:38.183813 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.184812 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.187040 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.187581 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.187708 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.188125 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:57:38.188250 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:57:38.188337 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:38.188521 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 12:57:38.188598 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 12:57:38.189043 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.181769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:32899 (local address 127.24.167.2:54413)
0808 12:57:38.182557 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:38.182596 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:38.182750 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:38.183133 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:38.183138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:38.183151 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 12:57:38.183646 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:38.183660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.184939 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.184947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:38.186894 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:38.186907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.187733 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.187739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:38.187901 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.187956 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 12:57:38.188337 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 12:57:38.188363 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 12:57:38.188471 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 12:57:38.188605 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":381,"threads_started":1}
I20260808 12:57:38.224699 3913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:38.225018 3913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:38.228581 3913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23: Sending DeleteTablet for 3 replicas of tablet f7a2f096f7f040cb951fd12627bc3473
I20260808 12:57:38.229406 4103 tablet_service.cc:1514] Processing DeleteTablet for tablet f7a2f096f7f040cb951fd12627bc3473 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:38 UTC) from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:60216
I20260808 12:57:38.229925 4526 tablet_service.cc:1514] Processing DeleteTablet for tablet f7a2f096f7f040cb951fd12627bc3473 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:38 UTC) from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:43180
I20260808 12:57:38.230324 4771 tablet_replica.cc:330] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f: stopping tablet replica
I20260808 12:57:38.230628 4771 raft_consensus.cc:2229] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:38.230990 4771 raft_consensus.cc:2258] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:38.231200 4323 tablet_service.cc:1514] Processing DeleteTablet for tablet f7a2f096f7f040cb951fd12627bc3473 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:38 UTC) from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:38552
I20260808 12:57:38.232264 4771 ts_tablet_manager.cc:1905] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:38.239651 4771 ts_tablet_manager.cc:1918] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 12:57:38.240304 4773 tablet_replica.cc:330] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707: stopping tablet replica
I20260808 12:57:38.240612 4773 raft_consensus.cc:2229] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:38.241010 4773 raft_consensus.cc:2258] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:38.242264 4773 ts_tablet_manager.cc:1905] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:38.249482 4771 log.cc:1196] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/f7a2f096f7f040cb951fd12627bc3473
I20260808 12:57:38.250139 4771 ts_tablet_manager.cc:1939] T f7a2f096f7f040cb951fd12627bc3473 P dcf66fd936fa42f5876a427ce101e74f: Deleting consensus metadata
I20260808 12:57:38.251641 3893 catalog_manager.cc:4841] TS dcf66fd936fa42f5876a427ce101e74f (127.24.167.1:36439): tablet f7a2f096f7f040cb951fd12627bc3473 (table test-table [id=f9c5d435e24e4f589658b8fb6aefb255]) successfully deleted
I20260808 12:57:38.252146 4773 ts_tablet_manager.cc:1918] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 12:57:38.252374 4773 log.cc:1196] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/f7a2f096f7f040cb951fd12627bc3473
I20260808 12:57:38.252943 4773 ts_tablet_manager.cc:1939] T f7a2f096f7f040cb951fd12627bc3473 P c9e48be2a62a45e2bcc413c65b39b707: Deleting consensus metadata
I20260808 12:57:38.254119 3891 catalog_manager.cc:4841] TS c9e48be2a62a45e2bcc413c65b39b707 (127.24.167.2:45851): tablet f7a2f096f7f040cb951fd12627bc3473 (table test-table [id=f9c5d435e24e4f589658b8fb6aefb255]) successfully deleted
I20260808 12:57:38.258045 4772 tablet_replica.cc:330] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef: stopping tablet replica
I20260808 12:57:38.258291 4772 raft_consensus.cc:2229] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:57:38.259316 4772 raft_consensus.cc:2258] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:38.260555 4772 ts_tablet_manager.cc:1905] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:38.268483 4772 ts_tablet_manager.cc:1918] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 12:57:38.268632 4772 log.cc:1196] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/f7a2f096f7f040cb951fd12627bc3473
I20260808 12:57:38.269133 4772 ts_tablet_manager.cc:1939] T f7a2f096f7f040cb951fd12627bc3473 P b65c90a0e0b049e488ed71d9f36a63ef: Deleting consensus metadata
I20260808 12:57:38.270430 3890 catalog_manager.cc:4841] TS b65c90a0e0b049e488ed71d9f36a63ef (127.24.167.3:32899): tablet f7a2f096f7f040cb951fd12627bc3473 (table test-table [id=f9c5d435e24e4f589658b8fb6aefb255]) successfully deleted
Aug 08 12:57:38 dist-test-slave-tk9z krb5kdc[3842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 12:57:38 dist-test-slave-tk9z krb5kdc[3842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:38 dist-test-slave-tk9z krb5kdc[3842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 12:57:38.326310 4730 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:38.312342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.24.167.62:40575)
0808 12:57:38.312701 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:38.312708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:38.313038 (+ 330us) server_negotiation.cc:407] Connection header received
0808 12:57:38.313241 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:38.313249 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:38.313349 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:38.313467 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:38.315396 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.316595 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.318551 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.319226 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.319388 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.326059 (+ 6671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37872: BlockingRecv error: recv got EOF from 127.0.0.1:37872 (error 108)
Metrics: {"server-negotiator.queue_time_us":181}
Aug 08 12:57:38 dist-test-slave-tk9z krb5kdc[3842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM
I20260808 12:57:38.387737 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.358007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37882 (local address 127.24.167.62:40575)
0808 12:57:38.358204 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:38.358211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:38.368838 (+ 10627us) server_negotiation.cc:407] Connection header received
0808 12:57:38.369312 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:38.369318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:38.369405 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:38.369476 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:38.371452 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.372597 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.374498 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.375120 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.375214 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.380800 (+ 5586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:38.380807 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:38.380813 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:38.380867 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:38.384447 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:38.385269 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:38.385275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:38.385279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:38.385437 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:38.385890 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:38.385896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:38.385899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:38.386278 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:38.386582 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:38.387111 (+ 529us) server_negotiation.cc:299] Negotiation successful
0808 12:57:38.387367 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 12:57:38.392751 3913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:57:38.393456 3913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:57:38.400704 4323 tablet_service.cc:1467] Processing CreateTablet for tablet 66e8eae07ba343769c09f12c92667beb (DEFAULT_TABLE table=test-table [id=0452488dc57b46d3b6051bdad509f738]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:38.401265 4323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e8eae07ba343769c09f12c92667beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:38.401242 4103 tablet_service.cc:1467] Processing CreateTablet for tablet 66e8eae07ba343769c09f12c92667beb (DEFAULT_TABLE table=test-table [id=0452488dc57b46d3b6051bdad509f738]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:38.401244 4526 tablet_service.cc:1467] Processing CreateTablet for tablet 66e8eae07ba343769c09f12c92667beb (DEFAULT_TABLE table=test-table [id=0452488dc57b46d3b6051bdad509f738]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:38.401774 4103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e8eae07ba343769c09f12c92667beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:38.401810 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e8eae07ba343769c09f12c92667beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:38.416443 4800 tablet_bootstrap.cc:492] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f: Bootstrap starting.
I20260808 12:57:38.418021 4805 tablet_bootstrap.cc:492] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707: Bootstrap starting.
I20260808 12:57:38.418290 4800 tablet_bootstrap.cc:654] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:38.419431 4804 tablet_bootstrap.cc:492] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef: Bootstrap starting.
I20260808 12:57:38.427380 4800 tablet_bootstrap.cc:492] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f: No bootstrap required, opened a new log
I20260808 12:57:38.427368 4805 tablet_bootstrap.cc:654] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:38.427553 4800 ts_tablet_manager.cc:1397] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260808 12:57:38.427580 4804 tablet_bootstrap.cc:654] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:38.428526 4800 raft_consensus.cc:348] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.428800 4800 raft_consensus.cc:374] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:38.428898 4800 raft_consensus.cc:729] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf66fd936fa42f5876a427ce101e74f, State: Initialized, Role: FOLLOWER
I20260808 12:57:38.429168 4800 consensus_queue.cc:260] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.429955 4800 ts_tablet_manager.cc:1428] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 12:57:38.431741 4804 tablet_bootstrap.cc:492] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef: No bootstrap required, opened a new log
I20260808 12:57:38.431910 4804 ts_tablet_manager.cc:1397] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260808 12:57:38.432842 4804 raft_consensus.cc:348] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.433135 4804 raft_consensus.cc:374] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:38.433254 4804 raft_consensus.cc:729] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65c90a0e0b049e488ed71d9f36a63ef, State: Initialized, Role: FOLLOWER
I20260808 12:57:38.433544 4804 consensus_queue.cc:260] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.434139 4804 ts_tablet_manager.cc:1428] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260808 12:57:38.443591 4805 tablet_bootstrap.cc:492] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707: No bootstrap required, opened a new log
I20260808 12:57:38.443780 4805 ts_tablet_manager.cc:1397] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.000s
I20260808 12:57:38.444744 4805 raft_consensus.cc:348] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.445044 4805 raft_consensus.cc:374] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:38.445160 4805 raft_consensus.cc:729] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e48be2a62a45e2bcc413c65b39b707, State: Initialized, Role: FOLLOWER
I20260808 12:57:38.445428 4805 consensus_queue.cc:260] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.446480 4805 ts_tablet_manager.cc:1428] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 12:57:38.572963 4682 raft_consensus.cc:482] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:57:38.573256 4682 raft_consensus.cc:504] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.575344 4682 leader_election.cc:288] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e48be2a62a45e2bcc413c65b39b707 (127.24.167.2:45851), b65c90a0e0b049e488ed71d9f36a63ef (127.24.167.3:32899)
I20260808 12:57:38.583040 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.575922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:38027 (local address 127.24.167.2:45851)
0808 12:57:38.576144 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:38.576150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:38.576616 (+ 466us) server_negotiation.cc:407] Connection header received
0808 12:57:38.576706 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:38.576713 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:38.576798 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:38.576903 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:38.577563 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.578572 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.580693 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.581962 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.582162 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.582253 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:38.582598 (+ 345us) server_negotiation.cc:299] Negotiation successful
0808 12:57:38.582753 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 12:57:38.583153 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.575525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:45851 (local address 127.24.167.1:38027)
0808 12:57:38.576142 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:38.576169 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:38.576346 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:38.577006 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:38.577010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:38.577037 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:38.577380 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:38.577389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.578710 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.578717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:38.580521 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:38.580536 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.582168 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.582175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:38.582395 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.582433 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 12:57:38.582547 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 12:57:38.582686 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":48,"thread_start_us":295,"threads_started":1}
I20260808 12:57:38.583922 4351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e8eae07ba343769c09f12c92667beb" candidate_uuid: "dcf66fd936fa42f5876a427ce101e74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e48be2a62a45e2bcc413c65b39b707" is_pre_election: true
I20260808 12:57:38.584208 4351 raft_consensus.cc:2454] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcf66fd936fa42f5876a427ce101e74f in term 0.
I20260808 12:57:38.584318 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.576343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:44453 (local address 127.24.167.3:32899)
0808 12:57:38.576562 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:38.576569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:38.577022 (+ 453us) server_negotiation.cc:407] Connection header received
0808 12:57:38.577301 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:38.577308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:38.577417 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:38.577514 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:57:38.578180 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.579344 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.581328 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.582753 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.582928 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.583016 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:38.583835 (+ 819us) server_negotiation.cc:299] Negotiation successful
0808 12:57:38.583988 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 12:57:38.584678 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.575546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:32899 (local address 127.24.167.1:44453)
0808 12:57:38.576925 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:38.576944 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:38.577138 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:38.577590 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:38.577594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:38.577622 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:57:38.578004 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:38.578016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.579505 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.579510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:38.581183 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:38.581194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.583509 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.583523 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:38.583645 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.583667 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 12:57:38.583792 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 12:57:38.583915 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1269,"mutex_wait_us":1,"thread_start_us":1097,"threads_started":1}
I20260808 12:57:38.584743 4037 leader_election.cc:302] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9e48be2a62a45e2bcc413c65b39b707, dcf66fd936fa42f5876a427ce101e74f; no voters:
I20260808 12:57:38.585317 4682 raft_consensus.cc:2781] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:57:38.585464 4682 raft_consensus.cc:482] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:57:38.585574 4682 raft_consensus.cc:3037] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:38.585389 4563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e8eae07ba343769c09f12c92667beb" candidate_uuid: "dcf66fd936fa42f5876a427ce101e74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" is_pre_election: true
I20260808 12:57:38.585765 4563 raft_consensus.cc:2454] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcf66fd936fa42f5876a427ce101e74f in term 0.
I20260808 12:57:38.588982 4682 raft_consensus.cc:504] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.589732 4682 leader_election.cc:288] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [CANDIDATE]: Term 1 election: Requested vote from peers c9e48be2a62a45e2bcc413c65b39b707 (127.24.167.2:45851), b65c90a0e0b049e488ed71d9f36a63ef (127.24.167.3:32899)
I20260808 12:57:38.590216 4351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e8eae07ba343769c09f12c92667beb" candidate_uuid: "dcf66fd936fa42f5876a427ce101e74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e48be2a62a45e2bcc413c65b39b707"
I20260808 12:57:38.590538 4351 raft_consensus.cc:3037] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:38.590435 4563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e8eae07ba343769c09f12c92667beb" candidate_uuid: "dcf66fd936fa42f5876a427ce101e74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65c90a0e0b049e488ed71d9f36a63ef"
I20260808 12:57:38.590708 4563 raft_consensus.cc:3037] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:38.593230 4563 raft_consensus.cc:2454] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcf66fd936fa42f5876a427ce101e74f in term 1.
I20260808 12:57:38.593628 4351 raft_consensus.cc:2454] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcf66fd936fa42f5876a427ce101e74f in term 1.
I20260808 12:57:38.593640 4036 leader_election.cc:302] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65c90a0e0b049e488ed71d9f36a63ef, dcf66fd936fa42f5876a427ce101e74f; no voters:
I20260808 12:57:38.593982 4682 raft_consensus.cc:2781] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:38.594528 4682 raft_consensus.cc:686] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [term 1 LEADER]: Becoming Leader. State: Replica: dcf66fd936fa42f5876a427ce101e74f, State: Running, Role: LEADER
I20260808 12:57:38.594985 4682 consensus_queue.cc:237] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.600768 3913 catalog_manager.cc:5495] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f reported cstate change: term changed from 0 to 1, leader changed from <none> to dcf66fd936fa42f5876a427ce101e74f (127.24.167.1). New cstate: current_term: 1 leader_uuid: "dcf66fd936fa42f5876a427ce101e74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } health_report { overall_health: UNKNOWN } } }
I20260808 12:57:38.621162 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.614413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.24.167.1:36439)
0808 12:57:38.615025 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:38.615032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:38.615056 (+ 24us) server_negotiation.cc:407] Connection header received
0808 12:57:38.615162 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:38.615167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:38.615286 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:38.615471 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:57:38.615832 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.617051 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.618346 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.619039 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.619176 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.619606 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:57:38.620436 (+ 830us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:57:38.620600 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:38.620673 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 12:57:38.620764 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":307,"threads_started":1}
I20260808 12:57:38.624924 4563 raft_consensus.cc:1261] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Refusing update from remote peer dcf66fd936fa42f5876a427ce101e74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:38.624967 4351 raft_consensus.cc:1261] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Refusing update from remote peer dcf66fd936fa42f5876a427ce101e74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:38.625771 4812 consensus_queue.cc:1035] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:38.626190 4682 consensus_queue.cc:1035] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:38.647203 3912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:38.647497 3912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:38.651485 3912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23: Sending DeleteTablet for 3 replicas of tablet 66e8eae07ba343769c09f12c92667beb
I20260808 12:57:38.652341 4103 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e8eae07ba343769c09f12c92667beb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:38 UTC) from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:60216
I20260808 12:57:38.652645 4323 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e8eae07ba343769c09f12c92667beb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:38 UTC) from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:38552
I20260808 12:57:38.652727 4771 tablet_replica.cc:330] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f: stopping tablet replica
I20260808 12:57:38.652941 4773 tablet_replica.cc:330] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707: stopping tablet replica
I20260808 12:57:38.652972 4771 raft_consensus.cc:2229] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:57:38.653153 4773 raft_consensus.cc:2229] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:38.653411 4773 raft_consensus.cc:2258] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:38.653920 4771 raft_consensus.cc:2258] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:38.654793 4526 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e8eae07ba343769c09f12c92667beb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:38 UTC) from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:43180
I20260808 12:57:38.655131 4772 tablet_replica.cc:330] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef: stopping tablet replica
I20260808 12:57:38.655156 4771 ts_tablet_manager.cc:1905] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:38.655328 4772 raft_consensus.cc:2229] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:38.655599 4772 raft_consensus.cc:2258] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:38.656998 4772 ts_tablet_manager.cc:1905] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:38.657647 3913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:57:38.658309 3913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:57:38.660241 4773 ts_tablet_manager.cc:1905] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:38.663985 4772 ts_tablet_manager.cc:1918] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:38.664162 4772 log.cc:1196] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/66e8eae07ba343769c09f12c92667beb
I20260808 12:57:38.664666 4772 ts_tablet_manager.cc:1939] T 66e8eae07ba343769c09f12c92667beb P b65c90a0e0b049e488ed71d9f36a63ef: Deleting consensus metadata
I20260808 12:57:38.665853 3890 catalog_manager.cc:4841] TS b65c90a0e0b049e488ed71d9f36a63ef (127.24.167.3:32899): tablet 66e8eae07ba343769c09f12c92667beb (table test-table [id=0452488dc57b46d3b6051bdad509f738]) successfully deleted
I20260808 12:57:38.667266 4103 tablet_service.cc:1467] Processing CreateTablet for tablet 51e5b45ca8664f8782d0e1130dbb0b01 (DEFAULT_TABLE table=test-table [id=cf93be9b317d4cb6925c7ed70b7ed654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:38.667738 4103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51e5b45ca8664f8782d0e1130dbb0b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:38.669565 4773 ts_tablet_manager.cc:1918] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:38.669721 4773 log.cc:1196] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/66e8eae07ba343769c09f12c92667beb
I20260808 12:57:38.670254 4773 ts_tablet_manager.cc:1939] T 66e8eae07ba343769c09f12c92667beb P c9e48be2a62a45e2bcc413c65b39b707: Deleting consensus metadata
I20260808 12:57:38.672185 4323 tablet_service.cc:1467] Processing CreateTablet for tablet 51e5b45ca8664f8782d0e1130dbb0b01 (DEFAULT_TABLE table=test-table [id=cf93be9b317d4cb6925c7ed70b7ed654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:38.672640 4323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51e5b45ca8664f8782d0e1130dbb0b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:38.676837 4800 tablet_bootstrap.cc:492] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f: Bootstrap starting.
I20260808 12:57:38.678737 4800 tablet_bootstrap.cc:654] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:38.678787 4526 tablet_service.cc:1467] Processing CreateTablet for tablet 51e5b45ca8664f8782d0e1130dbb0b01 (DEFAULT_TABLE table=test-table [id=cf93be9b317d4cb6925c7ed70b7ed654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:57:38.679234 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51e5b45ca8664f8782d0e1130dbb0b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:38.682215 4771 ts_tablet_manager.cc:1918] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:57:38.682822 4771 log.cc:1196] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/66e8eae07ba343769c09f12c92667beb
I20260808 12:57:38.683344 4771 ts_tablet_manager.cc:1939] T 66e8eae07ba343769c09f12c92667beb P dcf66fd936fa42f5876a427ce101e74f: Deleting consensus metadata
I20260808 12:57:38.684651 3893 catalog_manager.cc:4841] TS dcf66fd936fa42f5876a427ce101e74f (127.24.167.1:36439): tablet 66e8eae07ba343769c09f12c92667beb (table test-table [id=0452488dc57b46d3b6051bdad509f738]) successfully deleted
I20260808 12:57:38.686604 4804 tablet_bootstrap.cc:492] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef: Bootstrap starting.
I20260808 12:57:38.688557 4804 tablet_bootstrap.cc:654] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:38.689836 4805 tablet_bootstrap.cc:492] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707: Bootstrap starting.
I20260808 12:57:38.689862 3891 catalog_manager.cc:4841] TS c9e48be2a62a45e2bcc413c65b39b707 (127.24.167.2:45851): tablet 66e8eae07ba343769c09f12c92667beb (table test-table [id=0452488dc57b46d3b6051bdad509f738]) successfully deleted
I20260808 12:57:38.691988 4805 tablet_bootstrap.cc:654] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:38.692883 4800 tablet_bootstrap.cc:492] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f: No bootstrap required, opened a new log
I20260808 12:57:38.693075 4800 ts_tablet_manager.cc:1397] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20260808 12:57:38.694063 4800 raft_consensus.cc:348] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.694295 4805 tablet_bootstrap.cc:492] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707: No bootstrap required, opened a new log
I20260808 12:57:38.694391 4800 raft_consensus.cc:374] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:38.694495 4800 raft_consensus.cc:729] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf66fd936fa42f5876a427ce101e74f, State: Initialized, Role: FOLLOWER
I20260808 12:57:38.694676 4805 ts_tablet_manager.cc:1397] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 12:57:38.694775 4800 consensus_queue.cc:260] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.695341 4800 ts_tablet_manager.cc:1428] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 12:57:38.695629 4805 raft_consensus.cc:348] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.695955 4805 raft_consensus.cc:374] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:38.696084 4805 raft_consensus.cc:729] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e48be2a62a45e2bcc413c65b39b707, State: Initialized, Role: FOLLOWER
I20260808 12:57:38.696388 4805 consensus_queue.cc:260] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.699194 4805 ts_tablet_manager.cc:1428] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260808 12:57:38.704188 4804 tablet_bootstrap.cc:492] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef: No bootstrap required, opened a new log
I20260808 12:57:38.704452 4804 ts_tablet_manager.cc:1397] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.004s
I20260808 12:57:38.705413 4804 raft_consensus.cc:348] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.705706 4804 raft_consensus.cc:374] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:38.705806 4804 raft_consensus.cc:729] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65c90a0e0b049e488ed71d9f36a63ef, State: Initialized, Role: FOLLOWER
I20260808 12:57:38.706086 4804 consensus_queue.cc:260] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.706718 4804 ts_tablet_manager.cc:1428] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 12:57:38.725344 4682 raft_consensus.cc:482] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:57:38.725605 4682 raft_consensus.cc:504] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.726481 4682 leader_election.cc:288] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e48be2a62a45e2bcc413c65b39b707 (127.24.167.2:45851), b65c90a0e0b049e488ed71d9f36a63ef (127.24.167.3:32899)
I20260808 12:57:38.727692 4563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e5b45ca8664f8782d0e1130dbb0b01" candidate_uuid: "dcf66fd936fa42f5876a427ce101e74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" is_pre_election: true
I20260808 12:57:38.727984 4563 raft_consensus.cc:2454] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcf66fd936fa42f5876a427ce101e74f in term 0.
I20260808 12:57:38.728333 4351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e5b45ca8664f8782d0e1130dbb0b01" candidate_uuid: "dcf66fd936fa42f5876a427ce101e74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e48be2a62a45e2bcc413c65b39b707" is_pre_election: true
I20260808 12:57:38.728534 4036 leader_election.cc:302] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b65c90a0e0b049e488ed71d9f36a63ef, dcf66fd936fa42f5876a427ce101e74f; no voters:
I20260808 12:57:38.728603 4351 raft_consensus.cc:2454] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcf66fd936fa42f5876a427ce101e74f in term 0.
I20260808 12:57:38.728863 4682 raft_consensus.cc:2781] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:57:38.728984 4682 raft_consensus.cc:482] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:57:38.729077 4682 raft_consensus.cc:3037] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:38.732527 4682 raft_consensus.cc:504] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.733670 4351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e5b45ca8664f8782d0e1130dbb0b01" candidate_uuid: "dcf66fd936fa42f5876a427ce101e74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e48be2a62a45e2bcc413c65b39b707"
I20260808 12:57:38.733886 4351 raft_consensus.cc:3037] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:38.733764 4563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e5b45ca8664f8782d0e1130dbb0b01" candidate_uuid: "dcf66fd936fa42f5876a427ce101e74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65c90a0e0b049e488ed71d9f36a63ef"
I20260808 12:57:38.734055 4563 raft_consensus.cc:3037] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:38.736935 4351 raft_consensus.cc:2454] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcf66fd936fa42f5876a427ce101e74f in term 1.
I20260808 12:57:38.737193 4563 raft_consensus.cc:2454] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcf66fd936fa42f5876a427ce101e74f in term 1.
I20260808 12:57:38.737463 4037 leader_election.cc:302] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9e48be2a62a45e2bcc413c65b39b707, dcf66fd936fa42f5876a427ce101e74f; no voters:
I20260808 12:57:38.737757 4812 raft_consensus.cc:2781] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:38.738519 4682 leader_election.cc:288] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [CANDIDATE]: Term 1 election: Requested vote from peers c9e48be2a62a45e2bcc413c65b39b707 (127.24.167.2:45851), b65c90a0e0b049e488ed71d9f36a63ef (127.24.167.3:32899)
I20260808 12:57:38.739552 4812 raft_consensus.cc:686] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [term 1 LEADER]: Becoming Leader. State: Replica: dcf66fd936fa42f5876a427ce101e74f, State: Running, Role: LEADER
I20260808 12:57:38.739868 4812 consensus_queue.cc:237] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } }
I20260808 12:57:38.743356 3912 catalog_manager.cc:5495] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f reported cstate change: term changed from 0 to 1, leader changed from <none> to dcf66fd936fa42f5876a427ce101e74f (127.24.167.1). New cstate: current_term: 1 leader_uuid: "dcf66fd936fa42f5876a427ce101e74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf66fd936fa42f5876a427ce101e74f" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 } health_report { overall_health: UNKNOWN } } }
I20260808 12:57:38.791651 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.784004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:45807 (local address 127.24.167.2:45851)
0808 12:57:38.784185 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:38.784192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:38.784849 (+ 657us) server_negotiation.cc:407] Connection header received
0808 12:57:38.785036 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:38.785042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:38.785180 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:38.785282 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:57:38.785937 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.787121 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.789270 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.789945 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.790067 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.790676 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:57:38.790845 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:57:38.790933 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:38.791181 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 12:57:38.791271 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 12:57:38.792050 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.783899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:45851 (local address 127.24.167.1:45807)
0808 12:57:38.784709 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:38.784735 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:38.784891 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:38.785323 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:38.785328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:38.785341 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 12:57:38.785750 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:38.785762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.787272 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.787285 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:38.789108 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:38.789124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.790167 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.790173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:38.790348 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.790447 (+ 99us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 12:57:38.790954 (+ 507us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 12:57:38.790975 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 12:57:38.791116 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 12:57:38.791256 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":445,"threads_started":1}
I20260808 12:57:38.805264 4563 raft_consensus.cc:1261] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Refusing update from remote peer dcf66fd936fa42f5876a427ce101e74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:38.805882 4351 raft_consensus.cc:1261] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Refusing update from remote peer dcf66fd936fa42f5876a427ce101e74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:57:38.805936 4830 consensus_queue.cc:1035] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65c90a0e0b049e488ed71d9f36a63ef" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:38.806617 4812 consensus_queue.cc:1035] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e48be2a62a45e2bcc413c65b39b707" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:57:38.865589 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.854988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:57463 (local address 127.24.167.1:36439)
0808 12:57:38.855171 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:38.855177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:38.859020 (+ 3843us) server_negotiation.cc:407] Connection header received
0808 12:57:38.859203 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:38.859209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:38.859302 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:38.859412 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:57:38.859862 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.861055 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.863313 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.864037 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.864133 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.864714 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:57:38.864861 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:57:38.864954 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:38.865130 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 12:57:38.865234 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 12:57:38.865590 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.854973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:36439 (local address 127.24.167.2:57463)
0808 12:57:38.858895 (+ 3922us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:38.858919 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:38.859060 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:38.859447 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:38.859452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:38.859466 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 12:57:38.859707 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:38.859718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.861219 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.861225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:38.863140 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:38.863157 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.864296 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.864302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:38.864480 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.864527 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 12:57:38.864978 (+ 451us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 12:57:38.864990 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 12:57:38.865069 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 12:57:38.865215 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3725,"thread_start_us":3733,"threads_started":1}
I20260808 12:57:38.896083 3912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:38.896353 3912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:57:38.899703 3912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f4865941bc44c159ded4a459026dc23: Sending DeleteTablet for 3 replicas of tablet 51e5b45ca8664f8782d0e1130dbb0b01
I20260808 12:57:38.900606 4526 tablet_service.cc:1514] Processing DeleteTablet for tablet 51e5b45ca8664f8782d0e1130dbb0b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:38 UTC) from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:43180
I20260808 12:57:38.900942 4772 tablet_replica.cc:330] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef: stopping tablet replica
I20260808 12:57:38.901136 4772 raft_consensus.cc:2229] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:38.901368 4772 raft_consensus.cc:2258] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:38.902499 4103 tablet_service.cc:1514] Processing DeleteTablet for tablet 51e5b45ca8664f8782d0e1130dbb0b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:38 UTC) from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:60216
I20260808 12:57:38.902563 4772 ts_tablet_manager.cc:1905] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:38.902823 4771 tablet_replica.cc:330] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f: stopping tablet replica
I20260808 12:57:38.903015 4771 raft_consensus.cc:2229] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:57:38.903403 4771 raft_consensus.cc:2258] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:38.904914 4771 ts_tablet_manager.cc:1905] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:38.909840 4772 ts_tablet_manager.cc:1918] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 12:57:38.910046 4772 log.cc:1196] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/51e5b45ca8664f8782d0e1130dbb0b01
I20260808 12:57:38.910633 4772 ts_tablet_manager.cc:1939] T 51e5b45ca8664f8782d0e1130dbb0b01 P b65c90a0e0b049e488ed71d9f36a63ef: Deleting consensus metadata
I20260808 12:57:38.914076 3890 catalog_manager.cc:4841] TS b65c90a0e0b049e488ed71d9f36a63ef (127.24.167.3:32899): tablet 51e5b45ca8664f8782d0e1130dbb0b01 (table test-table [id=cf93be9b317d4cb6925c7ed70b7ed654]) successfully deleted
I20260808 12:57:38.919498 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 4009
I20260808 12:57:38.921969 4323 tablet_service.cc:1514] Processing DeleteTablet for tablet 51e5b45ca8664f8782d0e1130dbb0b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:57:38 UTC) from {username='oryx', principal='oryx/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:38552
I20260808 12:57:38.922695 4773 tablet_replica.cc:330] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707: stopping tablet replica
I20260808 12:57:38.922967 4773 raft_consensus.cc:2229] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:57:38.923257 4773 raft_consensus.cc:2258] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:57:38.923904 4771 ts_tablet_manager.cc:1918] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 12:57:38.924057 4771 log.cc:1196] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/51e5b45ca8664f8782d0e1130dbb0b01
I20260808 12:57:38.924535 4771 ts_tablet_manager.cc:1939] T 51e5b45ca8664f8782d0e1130dbb0b01 P dcf66fd936fa42f5876a427ce101e74f: Deleting consensus metadata
I20260808 12:57:38.924844 4773 ts_tablet_manager.cc:1905] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:57:38.925700 3893 catalog_manager.cc:4841] TS dcf66fd936fa42f5876a427ce101e74f (127.24.167.1:36439): tablet 51e5b45ca8664f8782d0e1130dbb0b01 (table test-table [id=cf93be9b317d4cb6925c7ed70b7ed654]) successfully deleted
I20260808 12:57:38.931905 4773 ts_tablet_manager.cc:1918] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 12:57:38.932057 4773 log.cc:1196] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/51e5b45ca8664f8782d0e1130dbb0b01
I20260808 12:57:38.932482 4773 ts_tablet_manager.cc:1939] T 51e5b45ca8664f8782d0e1130dbb0b01 P c9e48be2a62a45e2bcc413c65b39b707: Deleting consensus metadata
I20260808 12:57:38.933568 3891 catalog_manager.cc:4841] TS c9e48be2a62a45e2bcc413c65b39b707 (127.24.167.2:45851): tablet 51e5b45ca8664f8782d0e1130dbb0b01 (table test-table [id=cf93be9b317d4cb6925c7ed70b7ed654]) successfully deleted
Aug 08 12:57:38 dist-test-slave-tk9z krb5kdc[3842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.167.1@KRBTEST.COM
I20260808 12:57:38.945523 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:38.921157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.24.167.1:36439)
0808 12:57:38.921849 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:38.921855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:38.926480 (+ 4625us) server_negotiation.cc:407] Connection header received
0808 12:57:38.926754 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:38.926760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:38.926878 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:38.926962 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:38.928788 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.929994 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.931924 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:38.932572 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:38.932680 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:38.938078 (+ 5398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:38.938093 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:38.938114 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:38.938161 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:38.942423 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:38.943272 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:38.943282 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:38.943285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:38.943453 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:38.943887 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:38.943892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:38.943896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:38.944331 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:38.944650 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:38.944924 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 12:57:38.945140 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534}
I20260808 12:57:38.946563 4207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:39.073130 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 4009
I20260808 12:57:39.106738 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 4221
Aug 08 12:57:39 dist-test-slave-tk9z krb5kdc[3842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.167.2@KRBTEST.COM
I20260808 12:57:39.127490 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:39.107673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38586 (local address 127.24.167.2:45851)
0808 12:57:39.107883 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:39.107890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:39.107912 (+ 22us) server_negotiation.cc:407] Connection header received
0808 12:57:39.108025 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:39.108029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:39.108129 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:39.108252 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:39.109554 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:39.110282 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:39.111715 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:39.112164 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:39.112323 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:39.120694 (+ 8371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:39.120711 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:39.120730 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:39.120768 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:39.124069 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:39.125018 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:39.125036 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:39.125039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:39.125224 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:39.125674 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:39.125680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:39.125686 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:39.126085 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:39.126502 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:39.126833 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 12:57:39.127113 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 12:57:39.128224 4418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:39.277451 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 4221
I20260808 12:57:39.312695 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 4427
Aug 08 12:57:39 dist-test-slave-tk9z krb5kdc[3842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.167.3@KRBTEST.COM
I20260808 12:57:39.328393 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:39.313505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.24.167.3:32899)
0808 12:57:39.313992 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:39.313997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:39.314013 (+ 16us) server_negotiation.cc:407] Connection header received
0808 12:57:39.314070 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:39.314074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:39.314134 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:39.314225 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:39.315618 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:39.316373 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:39.317576 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:39.317969 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:39.318058 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:39.322606 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:39.322623 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:39.322640 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:39.322679 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:39.325699 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:39.326431 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:39.326444 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:39.326447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:39.326583 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:39.326936 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:39.326940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:39.326942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:39.327311 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:39.327538 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:39.327785 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 12:57:39.327959 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":277,"threads_started":1}
I20260808 12:57:39.329097 4620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:39.456617 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 4427
I20260808 12:57:39.484659 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 3863
I20260808 12:57:39.485991 3962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:39.625921 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 3863
2026-08-08T12:57:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7582 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:57:39 dist-test-slave-tk9z krb5kdc[4861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:57:39 dist-test-slave-tk9z krb5kdc[4861](info): set up 2 sockets
Aug 08 12:57:39 dist-test-slave-tk9z krb5kdc[4861](info): commencing operation
krb5kdc: starting...
W20260808 12:57:40.939890 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.259s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:57:41 dist-test-slave-tk9z krb5kdc[4861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 12:57:41.193979 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:34601
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:34601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:41.524894 4880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:41.525195 4880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:41.525362 4880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:41.532511 4880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:57:41.532613 4880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:41.532706 4880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:41.532817 4880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:57:41.532931 4880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:57:41.542580 4880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:34601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:34601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:41.543869 4880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:41.545169 4880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:41.557427 4890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:41.557533 4886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:41.557504 4887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:41.558113 4880 server_base.cc:1034] running on GCE node
I20260808 12:57:41.559003 4880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:41.560521 4880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:41.561765 4880 hybrid_clock.cc:648] HybridClock initialized: now 1786193861561702 us; error 55 us; skew 500 ppm
I20260808 12:57:41.562129 4880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T12:57:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1997 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:57:41 dist-test-slave-tk9z krb5kdc[4900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:57:41 dist-test-slave-tk9z krb5kdc[4900](info): set up 2 sockets
Aug 08 12:57:41 dist-test-slave-tk9z krb5kdc[4900](info): commencing operation
krb5kdc: starting...
W20260808 12:57:42.939025 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.260s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:57:43 dist-test-slave-tk9z krb5kdc[4900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:57:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 12:57:43.204308 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:45701
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:45701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:43.547405 4917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:43.547772 4917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:43.547959 4917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:43.555680 4917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:57:43.555814 4917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:43.555922 4917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:43.556026 4917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:57:43.556113 4917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:57:43.566429 4917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:45701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:45701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:43.567827 4917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:43.569202 4917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:43.582342 4923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:43.582423 4925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:43.582461 4922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:43.583328 4917 server_base.cc:1034] running on GCE node
I20260808 12:57:43.584105 4917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:43.585526 4917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:43.586786 4917 hybrid_clock.cc:648] HybridClock initialized: now 1786193863586744 us; error 56 us; skew 500 ppm
I20260808 12:57:43.587153 4917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:43 dist-test-slave-tk9z krb5kdc[4900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193863, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:43.590474 4917 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:57:43.592068 4917 webserver.cc:460] Webserver started at http://127.24.167.62:34319/ using document root <none> and password file <none>
I20260808 12:57:43.592772 4917 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:43.592880 4917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:43.593156 4917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:43.596211 4917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "60d25671aa704ed1a4d623021002e52b"
format_stamp: "Formatted at 2026-08-08 12:57:43 on dist-test-slave-tk9z"
server_key: "56367c9608071eecf81a4b23d376499d"
server_key_iv: "26461c55fc30f313649079234533d329"
server_key_version: "encryptionkey@0"
I20260808 12:57:43.596932 4917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "60d25671aa704ed1a4d623021002e52b"
format_stamp: "Formatted at 2026-08-08 12:57:43 on dist-test-slave-tk9z"
server_key: "56367c9608071eecf81a4b23d376499d"
server_key_iv: "26461c55fc30f313649079234533d329"
server_key_version: "encryptionkey@0"
I20260808 12:57:43.602062 4917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 12:57:43.605530 4933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:43.606946 4917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 12:57:43.607144 4917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "60d25671aa704ed1a4d623021002e52b"
format_stamp: "Formatted at 2026-08-08 12:57:43 on dist-test-slave-tk9z"
server_key: "56367c9608071eecf81a4b23d376499d"
server_key_iv: "26461c55fc30f313649079234533d329"
server_key_version: "encryptionkey@0"
I20260808 12:57:43.607337 4917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:43.620865 4917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:43.624624 4917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:43.624914 4917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:43.643810 4917 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:45701
I20260808 12:57:43.643821 5009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:45701 every 8 connection(s)
I20260808 12:57:43.645236 4917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:57:43.649024 5011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:57:43.650839 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 4917
I20260808 12:57:43.651101 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:57:43.651916 25244 external_mini_cluster.cc:1428] Setting key 7c1c56bc222d34c6d2306109f95c63b7
I20260808 12:57:43.664089 5011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b: Bootstrap starting.
Aug 08 12:57:43 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:43.667918 5011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b: Neither blocks nor log segments found. Creating new log.
I20260808 12:57:43.669147 5011 log.cc:824] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:43.672264 5011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b: No bootstrap required, opened a new log
I20260808 12:57:43.675704 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:43.653984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48054 (local address 127.24.167.62:45701)
0808 12:57:43.654824 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:43.654835 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:43.654886 (+ 51us) server_negotiation.cc:407] Connection header received
0808 12:57:43.656470 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:43.656496 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:43.656863 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:43.657176 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:43.659146 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:43.660424 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:43.662253 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:43.662950 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:43.663085 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:43.667737 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:43.667744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:43.667757 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:43.667792 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:43.671511 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:43.672203 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:43.672213 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:43.672227 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:43.672445 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:43.672905 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:43.672908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:43.672910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:43.673426 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:43.673649 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:43.674148 (+ 499us) server_negotiation.cc:299] Negotiation successful
0808 12:57:43.674620 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":406,"threads_started":1}
I20260808 12:57:43.681247 5011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d25671aa704ed1a4d623021002e52b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45701 } }
I20260808 12:57:43.681667 5011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:57:43.681813 5011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d25671aa704ed1a4d623021002e52b, State: Initialized, Role: FOLLOWER
I20260808 12:57:43.682531 5011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d25671aa704ed1a4d623021002e52b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45701 } }
I20260808 12:57:43.682760 5011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:57:43.682855 5011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:57:43.682983 5011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:57:43.685103 5011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d25671aa704ed1a4d623021002e52b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45701 } }
I20260808 12:57:43.685575 5011 leader_election.cc:302] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60d25671aa704ed1a4d623021002e52b; no voters:
I20260808 12:57:43.686153 5011 leader_election.cc:288] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:57:43.686303 5021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:57:43.687074 5021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 1 LEADER]: Becoming Leader. State: Replica: 60d25671aa704ed1a4d623021002e52b, State: Running, Role: LEADER
I20260808 12:57:43.687578 5021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d25671aa704ed1a4d623021002e52b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45701 } }
I20260808 12:57:43.688393 5011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:57:43.690609 5022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60d25671aa704ed1a4d623021002e52b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d25671aa704ed1a4d623021002e52b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45701 } } }
I20260808 12:57:43.690938 5022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:43.691604 5024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60d25671aa704ed1a4d623021002e52b. Latest consensus state: current_term: 1 leader_uuid: "60d25671aa704ed1a4d623021002e52b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d25671aa704ed1a4d623021002e52b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45701 } } }
I20260808 12:57:43.691817 5031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:57:43.691936 5024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:43.698482 5031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:57:43.707605 5031 catalog_manager.cc:1347] Generated new cluster ID: 549824cba41649d8938c3b189207d167
I20260808 12:57:43.707742 5031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:57:43.722777 5031 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:57:43.724176 5031 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:57:43.729723 5031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b: Generated new TSK 0
I20260808 12:57:43.730583 5031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 12:57:43.797631 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:45701
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:44.136431 5052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:44.136729 5052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:44.136901 5052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:44.144383 5052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:44.144505 5052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:44.144682 5052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:57:44.154448 5052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:45701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:44.155733 5052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:44.157024 5052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:44.169991 5066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:44.170059 5060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:44.170054 5061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:44.171188 5052 server_base.cc:1034] running on GCE node
I20260808 12:57:44.171837 5052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:44.172772 5052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:44.174032 5052 hybrid_clock.cc:648] HybridClock initialized: now 1786193864173982 us; error 55 us; skew 500 ppm
I20260808 12:57:44.174496 5052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:44 dist-test-slave-tk9z krb5kdc[4900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193864, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:44.181744 5052 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 12:57:44.183285 5052 webserver.cc:460] Webserver started at http://127.24.167.1:35715/ using document root <none> and password file <none>
I20260808 12:57:44.183996 5052 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:44.184149 5052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:44.184494 5052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:44.187590 5052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "c66cf7e9f2f541fbb5dd986ed986afb9"
format_stamp: "Formatted at 2026-08-08 12:57:44 on dist-test-slave-tk9z"
server_key: "e2b32848a124bce722fd29a9b108b89c"
server_key_iv: "a00ab13825189007819ffb55ea0e1795"
server_key_version: "encryptionkey@0"
I20260808 12:57:44.188407 5052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "c66cf7e9f2f541fbb5dd986ed986afb9"
format_stamp: "Formatted at 2026-08-08 12:57:44 on dist-test-slave-tk9z"
server_key: "e2b32848a124bce722fd29a9b108b89c"
server_key_iv: "a00ab13825189007819ffb55ea0e1795"
server_key_version: "encryptionkey@0"
I20260808 12:57:44.193729 5052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 12:57:44.197242 5074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:44.198616 5052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:57:44.198796 5052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "c66cf7e9f2f541fbb5dd986ed986afb9"
format_stamp: "Formatted at 2026-08-08 12:57:44 on dist-test-slave-tk9z"
server_key: "e2b32848a124bce722fd29a9b108b89c"
server_key_iv: "a00ab13825189007819ffb55ea0e1795"
server_key_version: "encryptionkey@0"
I20260808 12:57:44.199015 5052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:44.220592 5052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:44.224713 5052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:44.225024 5052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:44.226042 5052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:44.227428 5052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:44.227560 5052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:44.227716 5052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:44.227790 5052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:44.261523 5052 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:33099
I20260808 12:57:44.261574 5247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:33099 every 8 connection(s)
I20260808 12:57:44.262910 5052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:57:44.272796 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 5052
I20260808 12:57:44.273056 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:57:44.273759 25244 external_mini_cluster.cc:1428] Setting key c89902628b0e96cd08d703839b2292b6
Aug 08 12:57:44 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193864, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:44.284013 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:44.265529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:33381 (local address 127.24.167.62:45701)
0808 12:57:44.266002 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:44.266007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:44.266876 (+ 869us) server_negotiation.cc:407] Connection header received
0808 12:57:44.268868 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:44.268874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:44.268975 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:44.269106 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:44.270674 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:44.271398 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:44.272963 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:44.273438 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:44.273582 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:44.277930 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:44.277935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:44.277938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:44.277977 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:44.280664 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:44.281446 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:44.281457 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:44.281459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:44.281584 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:44.282081 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:44.282085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:44.282087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:44.282425 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:44.282602 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:44.283370 (+ 768us) server_negotiation.cc:299] Negotiation successful
0808 12:57:44.283630 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":245,"threads_started":1}
I20260808 12:57:44.285068 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:44.265909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:33381)
0808 12:57:44.266701 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:44.266741 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:44.268574 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:44.269359 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:44.269367 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:44.269883 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:44.270486 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:44.270504 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:44.271571 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:44.271584 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:44.272791 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:44.272803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:44.273697 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:44.273705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:44.273990 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:44.274725 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:44.274763 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:44.277687 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:44.280815 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:44.280825 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:44.280844 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:44.281304 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:44.281719 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:44.281724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:44.281728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:44.281942 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:44.282570 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:44.282579 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:44.282966 (+ 387us) client_negotiation.cc:769] Sending connection context
0808 12:57:44.283250 (+ 284us) client_negotiation.cc:240] Negotiation successful
0808 12:57:44.283850 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":270,"threads_started":1}
I20260808 12:57:44.286898 5248 heartbeater.cc:347] Connected to a master server at 127.24.167.62:45701
I20260808 12:57:44.287235 5248 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:44.288005 5248 heartbeater.cc:511] Master 127.24.167.62:45701 requested a full tablet report, sending...
I20260808 12:57:44.290503 4954 ts_manager.cc:194] Registered new tserver with Master: c66cf7e9f2f541fbb5dd986ed986afb9 (127.24.167.1:33099)
I20260808 12:57:44.292770 4954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:33381
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 12:57:44.334045 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:45701
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:44.680271 5259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:44.680569 5259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:44.680665 5259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:44.687963 5259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:44.688117 5259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:44.688264 5259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:57:44.698668 5259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:45701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:44.699908 5259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:44.701339 5259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:44.714725 5268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:44.714805 5271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:44.714776 5275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:44.715917 5259 server_base.cc:1034] running on GCE node
I20260808 12:57:44.716560 5259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:44.717456 5259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:44.718650 5259 hybrid_clock.cc:648] HybridClock initialized: now 1786193864718586 us; error 58 us; skew 500 ppm
I20260808 12:57:44.718976 5259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:44 dist-test-slave-tk9z krb5kdc[4900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193864, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:44.722105 5259 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 12:57:44.723663 5259 webserver.cc:460] Webserver started at http://127.24.167.2:33827/ using document root <none> and password file <none>
I20260808 12:57:44.724366 5259 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:44.724499 5259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:44.724830 5259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:44.727820 5259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "fe8be149858b410d8fbc6dca8211b9c1"
format_stamp: "Formatted at 2026-08-08 12:57:44 on dist-test-slave-tk9z"
server_key: "9835735cbef872795152f83c09d19bce"
server_key_iv: "0471a69620c7077bf85f5c021f4c45d0"
server_key_version: "encryptionkey@0"
I20260808 12:57:44.728618 5259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "fe8be149858b410d8fbc6dca8211b9c1"
format_stamp: "Formatted at 2026-08-08 12:57:44 on dist-test-slave-tk9z"
server_key: "9835735cbef872795152f83c09d19bce"
server_key_iv: "0471a69620c7077bf85f5c021f4c45d0"
server_key_version: "encryptionkey@0"
I20260808 12:57:44.733855 5259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 12:57:44.737462 5285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:44.738880 5259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:57:44.739074 5259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "fe8be149858b410d8fbc6dca8211b9c1"
format_stamp: "Formatted at 2026-08-08 12:57:44 on dist-test-slave-tk9z"
server_key: "9835735cbef872795152f83c09d19bce"
server_key_iv: "0471a69620c7077bf85f5c021f4c45d0"
server_key_version: "encryptionkey@0"
I20260808 12:57:44.739288 5259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:44.757697 5259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:44.761837 5259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:44.762151 5259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:44.763172 5259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:44.764612 5259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:44.764719 5259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:44.764843 5259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:44.764886 5259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:44.799293 5259 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:38601
I20260808 12:57:44.799340 5455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:38601 every 8 connection(s)
I20260808 12:57:44.800689 5259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:57:44.810722 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 5259
I20260808 12:57:44.810940 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:57:44.811398 25244 external_mini_cluster.cc:1428] Setting key b21f597694d258537b78d21623fbb1e4
Aug 08 12:57:44 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193864, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:44.821235 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:44.803533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:51601 (local address 127.24.167.62:45701)
0808 12:57:44.804045 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:44.804050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:44.804957 (+ 907us) server_negotiation.cc:407] Connection header received
0808 12:57:44.806755 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:44.806761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:44.806830 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:44.806929 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:44.808518 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:44.809235 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:44.810871 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:44.811376 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:44.811576 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:44.815495 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:44.815500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:44.815503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:44.815535 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:44.818112 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:44.818919 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:44.818927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:44.818935 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:44.819059 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:44.819503 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:44.819508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:44.819510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:44.819777 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:44.819940 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:44.820632 (+ 692us) server_negotiation.cc:299] Negotiation successful
0808 12:57:44.820857 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":284,"threads_started":1}
I20260808 12:57:44.822258 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:44.804108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:51601)
0808 12:57:44.804797 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:44.804837 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:44.806512 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:44.807143 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:44.807150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:44.807684 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:44.808261 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:44.808278 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:44.809367 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:44.809371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:44.810718 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:44.810731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:44.811724 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:44.811732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:44.812059 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:44.812776 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:44.812805 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:44.815264 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:44.818282 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:44.818291 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:44.818307 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:44.818793 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:44.819178 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:44.819183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:44.819187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:44.819373 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:44.819911 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:44.819919 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:44.820271 (+ 352us) client_negotiation.cc:769] Sending connection context
0808 12:57:44.820544 (+ 273us) client_negotiation.cc:240] Negotiation successful
0808 12:57:44.821089 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":270,"threads_started":1}
I20260808 12:57:44.823829 5462 heartbeater.cc:347] Connected to a master server at 127.24.167.62:45701
I20260808 12:57:44.824155 5462 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:44.825064 5462 heartbeater.cc:511] Master 127.24.167.62:45701 requested a full tablet report, sending...
I20260808 12:57:44.826745 4954 ts_manager.cc:194] Registered new tserver with Master: fe8be149858b410d8fbc6dca8211b9c1 (127.24.167.2:38601)
I20260808 12:57:44.828006 4954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:51601
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 12:57:44.871045 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:45701
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:45.214499 5469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:45.214845 5469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:45.214993 5469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:45.222167 5469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:45.222283 5469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:45.222461 5469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:57:45.232396 5469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:45701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:45.233657 5469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:45.235153 5469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:45.248813 5477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:45.248821 5478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:45.248842 5481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:45.249464 5469 server_base.cc:1034] running on GCE node
I20260808 12:57:45.250136 5469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:45.251192 5469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:45.252425 5469 hybrid_clock.cc:648] HybridClock initialized: now 1786193865252367 us; error 48 us; skew 500 ppm
I20260808 12:57:45.252780 5469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:45 dist-test-slave-tk9z krb5kdc[4900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193865, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:45.256271 5469 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 12:57:45.257827 5469 webserver.cc:460] Webserver started at http://127.24.167.3:37985/ using document root <none> and password file <none>
I20260808 12:57:45.258630 5469 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:45.258781 5469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:45.259106 5469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:57:45.262159 5469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "c162fff3e34d46fdbd6a828adb7301c4"
format_stamp: "Formatted at 2026-08-08 12:57:45 on dist-test-slave-tk9z"
server_key: "5c280d2061b78dadc05989e765ee0156"
server_key_iv: "1d427cab17b53772d8d4e23a3a964e6a"
server_key_version: "encryptionkey@0"
I20260808 12:57:45.263041 5469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "c162fff3e34d46fdbd6a828adb7301c4"
format_stamp: "Formatted at 2026-08-08 12:57:45 on dist-test-slave-tk9z"
server_key: "5c280d2061b78dadc05989e765ee0156"
server_key_iv: "1d427cab17b53772d8d4e23a3a964e6a"
server_key_version: "encryptionkey@0"
I20260808 12:57:45.268141 5469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 12:57:45.271538 5492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:45.272810 5469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 12:57:45.272962 5469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "c162fff3e34d46fdbd6a828adb7301c4"
format_stamp: "Formatted at 2026-08-08 12:57:45 on dist-test-slave-tk9z"
server_key: "5c280d2061b78dadc05989e765ee0156"
server_key_iv: "1d427cab17b53772d8d4e23a3a964e6a"
server_key_version: "encryptionkey@0"
I20260808 12:57:45.273129 5469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:45.296048 5248 heartbeater.cc:503] Master 127.24.167.62:45701 was elected leader, sending a full tablet report...
I20260808 12:57:45.296873 5469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:45.301098 5469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:45.301391 5469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:45.302389 5469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:45.303817 5469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:45.303920 5469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:45.304072 5469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:45.304157 5469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:45.336983 5469 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:38809
I20260808 12:57:45.337071 5669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:38809 every 8 connection(s)
I20260808 12:57:45.338279 5469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:57:45.347354 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 5469
I20260808 12:57:45.347556 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:57:45.347987 25244 external_mini_cluster.cc:1428] Setting key 7602270a4b9da787ea73a3cd4fc42b7c
Aug 08 12:57:45 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193865, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:57:45.358973 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:45.340866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:42667 (local address 127.24.167.62:45701)
0808 12:57:45.341319 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:45.341323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:45.342292 (+ 969us) server_negotiation.cc:407] Connection header received
0808 12:57:45.344472 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:45.344477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:45.344591 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:45.344718 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:45.346210 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:45.347179 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:45.349097 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:45.349716 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:45.349852 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:45.353440 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:45.353446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:45.353450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:45.353482 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:45.356190 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:45.356833 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:45.356837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:45.356839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:45.356972 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:45.357360 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:45.357363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:45.357365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:45.357615 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:45.357734 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:45.358541 (+ 807us) server_negotiation.cc:299] Negotiation successful
0808 12:57:45.358698 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":241,"threads_started":1}
I20260808 12:57:45.359903 5675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:45.341418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:42667)
0808 12:57:45.342123 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:45.342166 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:45.344197 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:45.344956 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:45.344964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:45.345477 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:45.346035 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:45.346052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:45.347346 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:45.347350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:45.348934 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:45.348951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:45.349981 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:45.349997 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:45.350262 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:45.351045 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:45.351080 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:45.353249 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:57:45.356314 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:45.356320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:45.356332 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:45.356735 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:45.357065 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:57:45.357068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:57:45.357070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:57:45.357241 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:57:45.357816 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:57:45.357822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:57:45.358111 (+ 289us) client_negotiation.cc:769] Sending connection context
0808 12:57:45.358480 (+ 369us) client_negotiation.cc:240] Negotiation successful
0808 12:57:45.358847 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":277,"threads_started":1}
I20260808 12:57:45.361132 5670 heartbeater.cc:347] Connected to a master server at 127.24.167.62:45701
I20260808 12:57:45.361426 5670 heartbeater.cc:464] Registering TS with master...
I20260808 12:57:45.362282 5670 heartbeater.cc:511] Master 127.24.167.62:45701 requested a full tablet report, sending...
I20260808 12:57:45.363968 4954 ts_manager.cc:194] Registered new tserver with Master: c162fff3e34d46fdbd6a828adb7301c4 (127.24.167.3:38809)
I20260808 12:57:45.364705 4954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:42667
I20260808 12:57:45.375669 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.1.keytab.
Entry for principal oryx/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.1.keytab.
WARNING: no policy specified for oryx/127.24.167.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.167.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.2.keytab.
Entry for principal oryx/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.2.keytab.
WARNING: no policy specified for oryx/127.24.167.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.167.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.3.keytab.
Entry for principal oryx/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.3.keytab.
I20260808 12:57:45.538582 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 5052
Aug 08 12:57:45 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:57:45.554744 5688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:45.539762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.24.167.1:33099)
0808 12:57:45.540441 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:45.540450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:45.540480 (+ 30us) server_negotiation.cc:407] Connection header received
0808 12:57:45.540553 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:45.540560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:45.540774 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:45.540918 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:45.542306 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:45.543498 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:45.544763 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:45.545318 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:45.545418 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:45.549366 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:45.549377 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:45.549392 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:45.549427 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:45.552284 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:45.552857 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:45.552864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:45.552867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:45.553007 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:45.553310 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:45.553313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:45.553315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:45.553665 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:45.553890 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:45.554114 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 12:57:45.554311 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":349,"threads_started":1}
I20260808 12:57:45.555982 5242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:45.639580 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 5052
I20260808 12:57:45.663748 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 5259
Aug 08 12:57:45 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 12:57:45.680435 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:45.664593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.24.167.2:38601)
0808 12:57:45.665147 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:45.665154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:45.665181 (+ 27us) server_negotiation.cc:407] Connection header received
0808 12:57:45.665243 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:45.665250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:45.665457 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:45.665585 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:45.667102 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:45.668140 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:45.669366 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:45.669917 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:45.670023 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:45.674503 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:45.674520 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:45.674548 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:45.674584 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:45.677528 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:45.678209 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:45.678216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:45.678218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:45.678345 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:45.678783 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:45.678788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:45.678792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:45.679141 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:45.679481 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:45.679789 (+ 308us) server_negotiation.cc:299] Negotiation successful
0808 12:57:45.680080 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":283,"threads_started":1}
I20260808 12:57:45.681715 5445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:45.763494 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 5259
I20260808 12:57:45.784281 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 5469
Aug 08 12:57:45 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 12:57:45.800791 5692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:45.785311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46390 (local address 127.24.167.3:38809)
0808 12:57:45.785842 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:45.785850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:45.785879 (+ 29us) server_negotiation.cc:407] Connection header received
0808 12:57:45.785951 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:45.785958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:45.786166 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:45.786300 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:45.787679 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:45.788717 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:45.790020 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:45.790656 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:45.790757 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:45.795101 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:45.795112 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:45.795127 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:45.795166 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:45.798239 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:45.798899 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:45.798907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:45.798909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:45.799055 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:45.799378 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:45.799381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:45.799383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:45.799732 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:45.799967 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:45.800181 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 12:57:45.800407 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":249,"threads_started":1}
I20260808 12:57:45.802083 5665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:45.889564 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 5469
I20260808 12:57:45.912382 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 4917
I20260808 12:57:45.913609 5000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:57:46.018220 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 4917
I20260808 12:57:46.049809 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:45701
--webserver_interface=127.24.167.62
--webserver_port=34319
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:45701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:57:46.390532 5696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:46.390834 5696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:46.391026 5696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:46.398360 5696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:57:46.398535 5696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:46.398627 5696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:46.398739 5696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:57:46.398828 5696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:57:46.408821 5696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:45701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:45701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=34319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:46.410307 5696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:46.411680 5696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:46.423612 5703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:46.423676 5701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:46.423859 5706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:46.424496 5696 server_base.cc:1034] running on GCE node
I20260808 12:57:46.425351 5696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:46.426833 5696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:46.428068 5696 hybrid_clock.cc:648] HybridClock initialized: now 1786193866428025 us; error 51 us; skew 500 ppm
I20260808 12:57:46.428412 5696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:46 dist-test-slave-tk9z krb5kdc[4900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193866, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:46.431524 5696 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:57:46.432966 5696 webserver.cc:460] Webserver started at http://127.24.167.62:34319/ using document root <none> and password file <none>
I20260808 12:57:46.433678 5696 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:46.433863 5696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:46.439339 5696 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 12:57:46.442041 5715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:46.443362 5696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 12:57:46.443555 5696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "60d25671aa704ed1a4d623021002e52b"
format_stamp: "Formatted at 2026-08-08 12:57:43 on dist-test-slave-tk9z"
server_key: "56367c9608071eecf81a4b23d376499d"
server_key_iv: "26461c55fc30f313649079234533d329"
server_key_version: "encryptionkey@0"
I20260808 12:57:46.444554 5696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:46.464617 5696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:46.469888 5696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:46.470173 5696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:46.489663 5696 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:45701
I20260808 12:57:46.489687 5786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:45701 every 8 connection(s)
I20260808 12:57:46.491323 5696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:57:46.495260 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 5696
I20260808 12:57:46.496054 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:33099
--local_ip_for_outbound_sockets=127.24.167.1
--tserver_master_addrs=127.24.167.62:45701
--webserver_port=35715
--webserver_interface=127.24.167.1
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.167.1
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 12:57:46.498812 5788 sys_catalog.cc:263] Verifying existing consensus state
I20260808 12:57:46.502137 5788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b: Bootstrap starting.
I20260808 12:57:46.521348 5788 log.cc:824] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b: Log is configured to *not* fsync() on all Append() calls
I20260808 12:57:46.530525 5788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 12:57:46.531086 5788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b: Bootstrap complete.
I20260808 12:57:46.540393 5788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d25671aa704ed1a4d623021002e52b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45701 } }
I20260808 12:57:46.541251 5788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d25671aa704ed1a4d623021002e52b, State: Initialized, Role: FOLLOWER
I20260808 12:57:46.541990 5788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d25671aa704ed1a4d623021002e52b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45701 } }
I20260808 12:57:46.542235 5788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:57:46.542407 5788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:57:46.542605 5788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 1 FOLLOWER]: Advancing to term 2
I20260808 12:57:46.546514 5788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d25671aa704ed1a4d623021002e52b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45701 } }
I20260808 12:57:46.547036 5788 leader_election.cc:302] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60d25671aa704ed1a4d623021002e52b; no voters:
I20260808 12:57:46.547713 5788 leader_election.cc:288] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 12:57:46.547909 5792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 2 FOLLOWER]: Leader election won for term 2
I20260808 12:57:46.549443 5792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [term 2 LEADER]: Becoming Leader. State: Replica: 60d25671aa704ed1a4d623021002e52b, State: Running, Role: LEADER
I20260808 12:57:46.550239 5788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:57:46.550122 5792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d25671aa704ed1a4d623021002e52b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45701 } }
I20260808 12:57:46.552888 5798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60d25671aa704ed1a4d623021002e52b. Latest consensus state: current_term: 2 leader_uuid: "60d25671aa704ed1a4d623021002e52b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d25671aa704ed1a4d623021002e52b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45701 } } }
I20260808 12:57:46.553282 5798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:46.553821 5794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "60d25671aa704ed1a4d623021002e52b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d25671aa704ed1a4d623021002e52b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45701 } } }
I20260808 12:57:46.554131 5794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b [sys.catalog]: This master's current role is: LEADER
I20260808 12:57:46.563549 5806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:57:46.569813 5806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:57:46.571553 5806 catalog_manager.cc:1259] Loaded cluster ID: 549824cba41649d8938c3b189207d167
I20260808 12:57:46.571743 5806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:57:46.575893 5806 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:57:46.577489 5806 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 60d25671aa704ed1a4d623021002e52b: Loaded TSK: 0
I20260808 12:57:46.579226 5806 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 12:57:46.855384 5790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:46.855731 5790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:46.855904 5790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:46.863492 5790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:46.863648 5790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:46.863832 5790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:57:46.874140 5790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.1.keytab
--principal=oryx/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:33099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=35715
--tserver_master_addrs=127.24.167.62:45701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:46.875595 5790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:46.876982 5790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:46.893481 5828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:46.894325 5827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:46.894850 5832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:46.895767 5790 server_base.cc:1034] running on GCE node
I20260808 12:57:46.896711 5790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:46.898142 5790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:46.899423 5790 hybrid_clock.cc:648] HybridClock initialized: now 1786193866899312 us; error 93 us; skew 500 ppm
I20260808 12:57:46.899847 5790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:46 dist-test-slave-tk9z krb5kdc[4900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193866, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:46.904275 5790 init.cc:377] Logged in from keytab as oryx/127.24.167.1@KRBTEST.COM (short username oryx)
I20260808 12:57:46.906080 5790 webserver.cc:460] Webserver started at http://127.24.167.1:35715/ using document root <none> and password file <none>
I20260808 12:57:46.907121 5790 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:46.907298 5790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:46.915530 5790 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.003s
I20260808 12:57:46.918754 5851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:46.920241 5790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 12:57:46.920437 5790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "c66cf7e9f2f541fbb5dd986ed986afb9"
format_stamp: "Formatted at 2026-08-08 12:57:44 on dist-test-slave-tk9z"
server_key: "e2b32848a124bce722fd29a9b108b89c"
server_key_iv: "a00ab13825189007819ffb55ea0e1795"
server_key_version: "encryptionkey@0"
I20260808 12:57:46.921495 5790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:46.937486 5790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:46.941881 5790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:46.942202 5790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:46.943331 5790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:46.944914 5790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:46.945057 5790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:46.945191 5790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:46.945281 5790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:46.981302 5790 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:33099
I20260808 12:57:46.981396 6015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:33099 every 8 connection(s)
I20260808 12:57:46.983002 5790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:57:46.983791 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 5790
I20260808 12:57:46.984570 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:38601
--local_ip_for_outbound_sockets=127.24.167.2
--tserver_master_addrs=127.24.167.62:45701
--webserver_port=33827
--webserver_interface=127.24.167.2
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.167.2
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 12:57:46 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:46 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 12:57:47.336896 6025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:47.337218 6025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:47.337379 6025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:47.344744 6025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:47.344898 6025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:47.345074 6025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:57:47.355270 6025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.2.keytab
--principal=oryx/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:38601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=33827
--tserver_master_addrs=127.24.167.62:45701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:47.356591 6025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:47.357872 6025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:47.375972 6033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:47.376329 6032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:47.377138 6025 server_base.cc:1034] running on GCE node
W20260808 12:57:47.377241 6036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:47.378077 6025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:47.379204 6025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:47.380441 6025 hybrid_clock.cc:648] HybridClock initialized: now 1786193867380388 us; error 43 us; skew 500 ppm
I20260808 12:57:47.380738 6025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:47 dist-test-slave-tk9z krb5kdc[4900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193867, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:47.384255 6025 init.cc:377] Logged in from keytab as oryx/127.24.167.2@KRBTEST.COM (short username oryx)
I20260808 12:57:47.385957 6025 webserver.cc:460] Webserver started at http://127.24.167.2:33827/ using document root <none> and password file <none>
I20260808 12:57:47.386699 6025 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:47.386821 6025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:47.392266 6025 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 12:57:47.395133 6043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:47.396556 6025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 12:57:47.396723 6025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "fe8be149858b410d8fbc6dca8211b9c1"
format_stamp: "Formatted at 2026-08-08 12:57:44 on dist-test-slave-tk9z"
server_key: "9835735cbef872795152f83c09d19bce"
server_key_iv: "0471a69620c7077bf85f5c021f4c45d0"
server_key_version: "encryptionkey@0"
I20260808 12:57:47.397691 6025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:47.416256 6025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:47.421099 6025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:47.421387 6025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:47.422505 6025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:47.424026 6025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:47.424137 6025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:47.424273 6025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:47.424362 6025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:47.467296 6025 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:38601
I20260808 12:57:47.467348 6219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:38601 every 8 connection(s)
I20260808 12:57:47.468978 6025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:57:47.476913 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 6025
I20260808 12:57:47.477861 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:38809
--local_ip_for_outbound_sockets=127.24.167.3
--tserver_master_addrs=127.24.167.62:45701
--webserver_port=37985
--webserver_interface=127.24.167.3
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.167.3
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 12:57:47 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:47 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 12:57:48.195668 6225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:57:48.196059 6225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:57:48.196236 6225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:57:48.204315 6225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:57:48.204465 6225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:57:48.204674 6225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:57:48.215087 6225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/krb5kdc/oryx_127.24.167.3.keytab
--principal=oryx/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:38809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=37985
--tserver_master_addrs=127.24.167.62:45701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:57:48.216910 6225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:57:48.218724 6225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:57:48.239605 6232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:48.242439 6233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:57:48.243690 6240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:57:48.243911 6225 server_base.cc:1034] running on GCE node
I20260808 12:57:48.244729 6225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:57:48.245923 6225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:57:48.247232 6225 hybrid_clock.cc:648] HybridClock initialized: now 1786193868247164 us; error 57 us; skew 500 ppm
I20260808 12:57:48.247645 6225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:57:48 dist-test-slave-tk9z krb5kdc[4900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193868, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:57:48.252436 6225 init.cc:377] Logged in from keytab as oryx/127.24.167.3@KRBTEST.COM (short username oryx)
I20260808 12:57:48.255928 6225 webserver.cc:460] Webserver started at http://127.24.167.3:37985/ using document root <none> and password file <none>
I20260808 12:57:48.256889 6225 fs_manager.cc:359] Metadata directory not provided
I20260808 12:57:48.257052 6225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:57:48.264935 6225 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 12:57:48.268702 6247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:48.270795 6225 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 12:57:48.270977 6225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "c162fff3e34d46fdbd6a828adb7301c4"
format_stamp: "Formatted at 2026-08-08 12:57:45 on dist-test-slave-tk9z"
server_key: "5c280d2061b78dadc05989e765ee0156"
server_key_iv: "1d427cab17b53772d8d4e23a3a964e6a"
server_key_version: "encryptionkey@0"
I20260808 12:57:48.272395 6225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:57:48.300376 6225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:57:48.305814 6225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:57:48.306128 6225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:57:48.307526 6225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:57:48.309327 6225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:57:48.309459 6225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:57:48.309562 6225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:57:48.309630 6225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260808 12:57:48.334791 6023 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.336s user 0.004s sys 0.011s
I20260808 12:57:48.337510 6023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:46.986555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:55347)
0808 12:57:46.987843 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:46.987899 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:46.989668 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:46.991394 (+ 1726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:46.991402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:46.991962 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:46.992513 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:46.992531 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:46.994904 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:46.994909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:46.996558 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:46.996577 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:46.997624 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:46.997633 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:46.997951 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:46.998774 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:46.998808 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:48.335990 (+1337182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":999,"thread_start_us":768,"threads_started":1}
W20260808 12:57:48.338086 6022 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:46.986563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:55347 (local address 127.24.167.62:45701)
0808 12:57:46.987382 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:46.987400 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:46.988046 (+ 646us) server_negotiation.cc:407] Connection header received
0808 12:57:46.990320 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:46.990342 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:46.990822 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:46.991282 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:46.992722 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:46.994715 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:46.996723 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:46.997375 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:46.997514 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:48.336780 (+1339266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:55347: BlockingRecv error: recv got EOF from 127.24.167.1:55347 (error 108)
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":304,"threads_started":1}
W20260808 12:57:48.338969 6016 heartbeater.cc:644] Failed to heartbeat to 127.24.167.62:45701 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.167.62:45701: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Aug 08 12:57:48 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:48 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:48.357419 6023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:48.339871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:58129)
0808 12:57:48.340918 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:48.340948 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:48.341137 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:48.341965 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:48.341974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:48.342712 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:48.343031 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:48.343048 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.345351 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.345360 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:48.347704 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:48.347727 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.349210 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.349220 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:48.349503 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:48.350497 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:48.350534 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:48.356746 (+ 6212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":726}
W20260808 12:57:48.357491 6022 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:48.340269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:58129 (local address 127.24.167.62:45701)
0808 12:57:48.340563 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:48.340575 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:48.341289 (+ 714us) server_negotiation.cc:407] Connection header received
0808 12:57:48.341389 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:48.341398 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:48.341515 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:48.341640 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:48.343410 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.345080 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.347962 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.348891 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.349953 (+ 1062us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:48.357129 (+ 7176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:58129: BlockingRecv error: recv got EOF from 127.24.167.1:58129 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 08 12:57:48 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:48 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:48.374264 6023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:48.358577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:37113)
0808 12:57:48.359085 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:48.359114 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:48.359265 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:48.359997 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:48.360007 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:48.360536 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:48.360832 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:48.360851 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.362799 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.362817 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:48.365143 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:48.365165 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.366824 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.366835 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:48.367150 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:48.368091 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:48.368125 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:48.373800 (+ 5675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326}
W20260808 12:57:48.374615 6022 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:48.359006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:37113 (local address 127.24.167.62:45701)
0808 12:57:48.359198 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:48.359206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:48.359229 (+ 23us) server_negotiation.cc:407] Connection header received
0808 12:57:48.359444 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:48.359454 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:48.359584 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:48.359716 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:48.361042 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.362556 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.365353 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.366356 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.366557 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:48.374189 (+ 7632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:37113: BlockingRecv error: recv got EOF from 127.24.167.1:37113 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260808 12:57:48.375305 6016 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 12:57:48.380146 6225 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:38809
I20260808 12:57:48.380213 6427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:38809 every 8 connection(s)
I20260808 12:57:48.381762 6225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:57:48.385638 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 6225
Aug 08 12:57:48 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:48 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:48.423451 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:48.386698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48060 (local address 127.24.167.62:45701)
0808 12:57:48.399148 (+ 12450us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:48.399158 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:48.401191 (+ 2033us) server_negotiation.cc:407] Connection header received
0808 12:57:48.401361 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:48.401371 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:48.401504 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:48.401674 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:48.404235 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.406122 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.408323 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.409276 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.409444 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:48.413401 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:57:48.413418 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:57:48.413454 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:57:48.413532 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:57:48.418464 (+ 4932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:48.419736 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:48.419748 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:48.419761 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:48.420001 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:57:48.420556 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:57:48.420564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:57:48.420570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:57:48.421374 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:57:48.421720 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0808 12:57:48.422311 (+ 591us) server_negotiation.cc:299] Negotiation successful
0808 12:57:48.422927 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12246,"thread_start_us":12473,"threads_started":1}
W20260808 12:57:48.674733 6224 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.185s user 0.007s sys 0.011s
W20260808 12:57:48.676174 6222 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:47.472636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:58665 (local address 127.24.167.62:45701)
0808 12:57:47.479665 (+ 7029us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:47.479675 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:47.479705 (+ 30us) server_negotiation.cc:407] Connection header received
0808 12:57:47.479800 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:47.479809 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:47.479926 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:47.480110 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:47.482398 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:47.484096 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:47.486801 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:47.487863 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:47.488062 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:48.675835 (+1187773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:58665: BlockingRecv error: recv got EOF from 127.24.167.2:58665 (error 108)
Metrics: {"server-negotiator.queue_time_us":6840,"thread_start_us":6660,"threads_started":1}
I20260808 12:57:48.676759 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:47.473420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:58665)
0808 12:57:47.474695 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:47.474771 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:47.478641 (+ 3870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:47.480406 (+ 1765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:47.480420 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:47.481251 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:47.482113 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:47.482148 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:47.484339 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:47.484347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:47.486595 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:47.486613 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:47.488156 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:47.488161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:47.488391 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:47.489251 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:47.489281 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:48.675603 (+1186322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":450,"threads_started":1}
W20260808 12:57:48.678478 6220 heartbeater.cc:644] Failed to heartbeat to 127.24.167.62:45701 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.167.62:45701: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Aug 08 12:57:48 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:48 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:48.689108 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:48.679577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:49665)
0808 12:57:48.679777 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:48.679796 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:48.679902 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:48.680493 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:48.680498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:48.680908 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:48.681076 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:48.681085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.682156 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.682161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:48.683182 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:48.683195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.683931 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.683938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:48.684104 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:48.684769 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:48.684787 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:48.688796 (+ 4009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
W20260808 12:57:48.689244 6222 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:48.680056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:49665 (local address 127.24.167.62:45701)
0808 12:57:48.680220 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:48.680225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:48.680240 (+ 15us) server_negotiation.cc:407] Connection header received
0808 12:57:48.680289 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:48.680293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:48.680349 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:48.680459 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:48.681224 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.682003 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.683327 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.683742 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.683898 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:48.688996 (+ 5098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:49665: BlockingRecv error: recv got EOF from 127.24.167.2:49665 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 12:57:48 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:48 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 12:57:48.703452 6222 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:48.690663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:41467 (local address 127.24.167.62:45701)
0808 12:57:48.690898 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:48.690907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:48.690928 (+ 21us) server_negotiation.cc:407] Connection header received
0808 12:57:48.690994 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:48.691000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:48.691096 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:48.691206 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:48.692466 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.693297 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.696150 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.696559 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.696662 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:48.703238 (+ 6576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:41467: BlockingRecv error: recv got EOF from 127.24.167.2:41467 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 12:57:48.703497 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:48.690292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:41467)
0808 12:57:48.690529 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:48.690549 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:48.690643 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:48.691304 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:48.691313 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:48.691940 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:48.692241 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:48.692258 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.693604 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.693615 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:48.695944 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:48.695968 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.696759 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.696769 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:48.697057 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:48.697859 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:48.697896 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:48.703079 (+ 5183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123}
W20260808 12:57:48.703936 6220 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 12:57:49 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:49 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:49.391805 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:49.376104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:34769)
0808 12:57:49.376927 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:49.376943 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:49.377102 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:49.377510 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:49.377514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:49.378100 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:49.378251 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:49.378260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.380076 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.380082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:49.381290 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:49.381303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.382051 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.382057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:49.382238 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:49.387681 (+ 5443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:49.387701 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:49.391467 (+ 3766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":549,"threads_started":1}
W20260808 12:57:49.393023 6439 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:49.376172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:34769 (local address 127.24.167.62:45701)
0808 12:57:49.376816 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:49.376824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:49.377028 (+ 204us) server_negotiation.cc:407] Connection header received
0808 12:57:49.377239 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:49.377250 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:49.377354 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:49.377471 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:49.378429 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.379873 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.381442 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.381858 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.381993 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:49.392750 (+ 10757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:34769: BlockingRecv error: recv got EOF from 127.24.167.1:34769 (error 108)
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":293,"threads_started":1}
W20260808 12:57:49.607015 6433 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.194s user 0.004s sys 0.014s
W20260808 12:57:49.608608 6022 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:48.384961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:60203 (local address 127.24.167.62:45701)
0808 12:57:48.385172 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:48.385177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:48.400361 (+ 15184us) server_negotiation.cc:407] Connection header received
0808 12:57:48.404470 (+ 4109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:48.404476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:48.404570 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:48.404707 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:48.406483 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.408095 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.410099 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.410772 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.410908 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:49.608324 (+1197416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:60203: BlockingRecv error: recv got EOF from 127.24.167.3:60203 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":74}
I20260808 12:57:49.609470 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:48.386059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:60203)
0808 12:57:48.400099 (+ 14040us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:48.400179 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:48.404140 (+ 3961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:48.405029 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:48.405039 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:48.405592 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:48.406208 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:48.406259 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.408331 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.408335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:48.409912 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:48.409931 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:48.411234 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:48.411243 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:48.411637 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:48.412497 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:48.412539 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:49.608120 (+1195581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13555,"thread_start_us":13253,"threads_started":1}
W20260808 12:57:49.610894 6429 heartbeater.cc:644] Failed to heartbeat to 127.24.167.62:45701 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.167.62:45701: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Aug 08 12:57:49 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:49 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:49.624272 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:49.611634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:46931)
0808 12:57:49.611847 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:49.611868 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:49.611980 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:49.612313 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:49.612316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:49.612866 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:49.613132 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:49.613146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.614772 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.614781 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:49.616405 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:49.616424 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.617927 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.617934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:49.618181 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:49.619440 (+ 1259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:49.619465 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:49.623967 (+ 4502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260808 12:57:49.624451 6022 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:49.611726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:46931 (local address 127.24.167.62:45701)
0808 12:57:49.611973 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:49.611980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:49.612005 (+ 25us) server_negotiation.cc:407] Connection header received
0808 12:57:49.612070 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:49.612076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:49.612167 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:49.612304 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:49.613308 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.614612 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.616577 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.617429 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.617574 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:49.624187 (+ 6613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:46931: BlockingRecv error: recv got EOF from 127.24.167.3:46931 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Aug 08 12:57:49 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:49 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:49.636299 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:49.624989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:46671)
0808 12:57:49.625175 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:49.625195 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:49.625309 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:49.625750 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:49.625757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:49.626229 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:49.626498 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:49.626514 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.628155 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.628164 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:49.629956 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:49.629976 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.631268 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.631273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:49.631490 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:49.632227 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:49.632244 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:49.636028 (+ 3784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260808 12:57:49.636456 6022 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:49.625027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:46671 (local address 127.24.167.62:45701)
0808 12:57:49.625194 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:49.625199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:49.625246 (+ 47us) server_negotiation.cc:407] Connection header received
0808 12:57:49.625444 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:49.625449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:49.625556 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:49.625661 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:49.626639 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.627955 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.630110 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.630925 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.631053 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:49.636213 (+ 5160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:46671: BlockingRecv error: recv got EOF from 127.24.167.3:46671 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 12:57:49.636703 6429 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 12:57:49 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:49 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:49.713845 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:49.704458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:56837)
0808 12:57:49.705013 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:49.705028 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:49.705191 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:49.705576 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:49.705582 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:49.706052 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:49.706203 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:49.706211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.707289 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.707300 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:49.708287 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:49.708299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.709098 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.709104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:49.709273 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:49.709841 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:49.709859 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:49.713491 (+ 3632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":324,"threads_started":1}
W20260808 12:57:49.714027 6022 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:49.704549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:56837 (local address 127.24.167.62:45701)
0808 12:57:49.704793 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:49.704802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:49.705126 (+ 324us) server_negotiation.cc:407] Connection header received
0808 12:57:49.705316 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:49.705322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:49.705409 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:49.705523 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:49.706355 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.707136 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.708440 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:49.708901 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:49.709036 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:49.713720 (+ 4684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:56837: BlockingRecv error: recv got EOF from 127.24.167.2:56837 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 08 12:57:50 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:50 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:50.402328 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:50.392906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:34761)
0808 12:57:50.393487 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:50.393504 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:50.393610 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:50.393909 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:50.393912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:50.394322 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:50.394544 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:50.394553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:50.395628 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:50.395632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:50.396544 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:50.396555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:50.397246 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:50.397250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:50.397370 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:50.397987 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:50.398004 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:50.402019 (+ 4015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":317,"threads_started":1}
W20260808 12:57:50.402580 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:50.393036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:34761 (local address 127.24.167.62:45701)
0808 12:57:50.393594 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:50.393598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:50.393615 (+ 17us) server_negotiation.cc:407] Connection header received
0808 12:57:50.393698 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:50.393702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:50.393760 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:50.393869 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:50.394673 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:50.395499 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:50.396681 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:50.397108 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:50.397210 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:50.402266 (+ 5056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:34761: BlockingRecv error: recv got EOF from 127.24.167.1:34761 (error 108)
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":348,"threads_started":1}
Aug 08 12:57:50 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:50 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:50.646817 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:50.637429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:58447)
0808 12:57:50.637966 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:50.637982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:50.638102 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:50.638486 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:50.638490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:50.638910 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:50.639086 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:50.639094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:50.640427 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:50.640433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:50.641577 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:50.641589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:50.642269 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:50.642273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:50.642414 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:50.642980 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:50.642997 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:50.646579 (+ 3582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":298,"threads_started":1}
W20260808 12:57:50.647125 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:50.637542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:58447 (local address 127.24.167.62:45701)
0808 12:57:50.637788 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:50.637794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:50.638069 (+ 275us) server_negotiation.cc:407] Connection header received
0808 12:57:50.638231 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:50.638236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:50.638299 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:50.638438 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:50.639213 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:50.640297 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:50.641704 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:50.642112 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:50.642217 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:50.646815 (+ 4598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:58447: BlockingRecv error: recv got EOF from 127.24.167.3:58447 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 08 12:57:50 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:50 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:50.725021 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:50.714840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:44345)
0808 12:57:50.715427 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:50.715444 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:50.715565 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:50.715863 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:50.715866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:50.716287 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:50.716480 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:50.716488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:50.717497 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:50.717502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:50.718625 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:50.718640 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:50.719349 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:50.719355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:50.719527 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:50.720135 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:50.720155 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:50.724509 (+ 4354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":333,"threads_started":1}
W20260808 12:57:50.725066 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:50.715060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:44345 (local address 127.24.167.62:45701)
0808 12:57:50.715240 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:50.715244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:50.715520 (+ 276us) server_negotiation.cc:407] Connection header received
0808 12:57:50.715660 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:50.715664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:50.715723 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:50.715806 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:50.716599 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:50.717322 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:50.718768 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:50.719174 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:50.719270 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:50.724778 (+ 5508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:44345: BlockingRecv error: recv got EOF from 127.24.167.2:44345 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 12:57:51 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:51 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:51.412637 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:51.403526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:59639)
0808 12:57:51.404124 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:51.404152 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:51.404262 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:51.404569 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:51.404572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:51.404932 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:51.405114 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:51.405122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:51.406115 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:51.406119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:51.407126 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:51.407137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:51.407793 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:51.407797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:51.407915 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:51.408484 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:51.408501 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:51.412361 (+ 3860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":288,"threads_started":1}
W20260808 12:57:51.412868 6454 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:51.403740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:59639 (local address 127.24.167.62:45701)
0808 12:57:51.404256 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:51.404260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:51.404277 (+ 17us) server_negotiation.cc:407] Connection header received
0808 12:57:51.404357 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:51.404360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:51.404417 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:51.404511 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:51.405248 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:51.405998 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:51.407260 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:51.407676 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:51.407764 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:51.412583 (+ 4819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:59639: BlockingRecv error: recv got EOF from 127.24.167.1:59639 (error 108)
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":269,"threads_started":1}
Aug 08 12:57:51 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:51 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:51.658195 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:51.647876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:60083)
0808 12:57:51.648434 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:51.648449 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:51.648609 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:51.648997 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:51.649003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:51.649516 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:51.649730 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:51.649744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:51.651058 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:51.651064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:51.652668 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:51.652681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:51.653357 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:51.653361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:51.653488 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:51.654114 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:51.654134 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:51.657950 (+ 3816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":359,"threads_started":1}
W20260808 12:57:51.658385 6454 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:51.647909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:60083 (local address 127.24.167.62:45701)
0808 12:57:51.648075 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:51.648080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:51.648535 (+ 455us) server_negotiation.cc:407] Connection header received
0808 12:57:51.648736 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:51.648744 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:51.648831 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:51.648939 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:51.649900 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:51.650906 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:51.652816 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:51.653224 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:51.653324 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:51.658152 (+ 4828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:60083: BlockingRecv error: recv got EOF from 127.24.167.3:60083 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 12:57:51 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:51 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:51.736413 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:51.726178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:41039)
0808 12:57:51.726803 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:51.726821 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:51.727014 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:51.727427 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:51.727432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:51.727856 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:51.728065 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:51.728076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:51.729282 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:51.729288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:51.730505 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:51.730520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:51.731379 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:51.731385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:51.731601 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:51.732229 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:51.732249 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:51.736175 (+ 3926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":332,"threads_started":1}
W20260808 12:57:51.736678 6454 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:51.726355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:41039 (local address 127.24.167.62:45701)
0808 12:57:51.726629 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:51.726637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:51.726916 (+ 279us) server_negotiation.cc:407] Connection header received
0808 12:57:51.727161 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:51.727167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:51.727262 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:51.727380 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:51.728240 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:51.729094 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:51.730667 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:51.731184 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:51.731348 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:51.736416 (+ 5068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:41039: BlockingRecv error: recv got EOF from 127.24.167.2:41039 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 08 12:57:52 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:52 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:52.423202 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:52.413727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:54087)
0808 12:57:52.414236 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:52.414252 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:52.414361 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:52.414870 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:52.414874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:52.415258 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:52.415425 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:52.415433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:52.416403 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:52.416406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:52.417383 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:52.417394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:52.418048 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:52.418053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:52.418180 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:52.418771 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:52.418789 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:52.422848 (+ 4059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":273,"threads_started":1}
W20260808 12:57:52.423738 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:52.413807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:54087 (local address 127.24.167.62:45701)
0808 12:57:52.414589 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:52.414593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:52.414608 (+ 15us) server_negotiation.cc:407] Connection header received
0808 12:57:52.414654 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:52.414658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:52.414713 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:52.414809 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:52.415554 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:52.416279 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:52.417513 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:52.417890 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:52.417990 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:52.423171 (+ 5181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:54087: BlockingRecv error: recv got EOF from 127.24.167.1:54087 (error 108)
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":560,"threads_started":1}
Aug 08 12:57:52 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:52 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:52.669627 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:52.659207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:33399)
0808 12:57:52.659828 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:52.659846 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:52.660034 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:52.660439 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:52.660446 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:52.660877 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:52.661112 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:52.661122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:52.662297 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:52.662304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:52.663559 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:52.663574 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:52.664529 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:52.664536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:52.664745 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:52.665422 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:52.665441 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:52.669248 (+ 3807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":336,"threads_started":1}
W20260808 12:57:52.669744 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:52.659341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:33399 (local address 127.24.167.62:45701)
0808 12:57:52.659606 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:52.659616 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:52.659953 (+ 337us) server_negotiation.cc:407] Connection header received
0808 12:57:52.660186 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:52.660191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:52.660268 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:52.660396 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:52.661298 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:52.662135 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:52.663718 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:52.664306 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:52.664500 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:52.669492 (+ 4992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:33399: BlockingRecv error: recv got EOF from 127.24.167.3:33399 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 08 12:57:52 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:52 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:52.747149 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:52.737472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:37993)
0808 12:57:52.738059 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:52.738076 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:52.738239 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:52.738649 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:52.738654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:52.739087 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:52.739255 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:52.739265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:52.740339 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:52.740346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:52.741339 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:52.741351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:52.742106 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:52.742112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:52.742305 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:52.742996 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:52.743013 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:52.746856 (+ 3843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":338,"threads_started":1}
W20260808 12:57:52.747377 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:52.737478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:37993 (local address 127.24.167.62:45701)
0808 12:57:52.737665 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:52.737674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:52.738170 (+ 496us) server_negotiation.cc:407] Connection header received
0808 12:57:52.738391 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:52.738397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:52.738485 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:52.738592 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:52.739413 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:52.740198 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:52.741502 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:52.741952 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:52.742090 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:52.747090 (+ 5000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:37993: BlockingRecv error: recv got EOF from 127.24.167.2:37993 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 12:57:53 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:53 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:53.433368 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:53.424202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:49409)
0808 12:57:53.424760 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:53.424777 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:53.424882 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:53.425189 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:53.425192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:53.425658 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:53.425882 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:53.425893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:53.427038 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:53.427042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:53.427994 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:53.428006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:53.428665 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:53.428669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:53.428811 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:53.429502 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:53.429520 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:53.433085 (+ 3565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":304,"threads_started":1}
W20260808 12:57:53.433794 6464 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:53.424270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:49409 (local address 127.24.167.62:45701)
0808 12:57:53.424836 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:53.424841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:53.424857 (+ 16us) server_negotiation.cc:407] Connection header received
0808 12:57:53.424989 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:53.424993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:53.425052 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:53.425143 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:53.426032 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:53.426910 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:53.428121 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:53.428542 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:53.428631 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:53.433450 (+ 4819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:49409: BlockingRecv error: recv got EOF from 127.24.167.1:49409 (error 108)
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":351,"threads_started":1}
Aug 08 12:57:53 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:53 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:53.680362 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:53.670733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:52429)
0808 12:57:53.671328 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:53.671344 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:53.671513 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:53.671868 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:53.671873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:53.672316 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:53.672480 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:53.672489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:53.673551 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:53.673556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:53.674630 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:53.674642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:53.675345 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:53.675351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:53.675545 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:53.676231 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:53.676252 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:53.680106 (+ 3854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":348,"threads_started":1}
W20260808 12:57:53.680663 6464 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:53.670835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:52429 (local address 127.24.167.62:45701)
0808 12:57:53.671089 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:53.671097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:53.671438 (+ 341us) server_negotiation.cc:407] Connection header received
0808 12:57:53.671647 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:53.671654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:53.671722 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:53.671828 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:53.672637 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:53.673395 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:53.674776 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:53.675190 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:53.675331 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:53.680368 (+ 5037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:52429: BlockingRecv error: recv got EOF from 127.24.167.3:52429 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 08 12:57:53 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:53 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:53.758165 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:53.748363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:56941)
0808 12:57:53.748968 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:53.748985 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:53.749154 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:53.749555 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:53.749560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:53.750043 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:53.750196 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:53.750205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:53.751521 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:53.751526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:53.752954 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:53.752967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:53.753760 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:53.753764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:53.753885 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:53.754579 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:53.754597 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:53.757924 (+ 3327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":325,"threads_started":1}
W20260808 12:57:53.758563 6464 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:53.748524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:56941 (local address 127.24.167.62:45701)
0808 12:57:53.748714 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:53.748719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:53.749082 (+ 363us) server_negotiation.cc:407] Connection header received
0808 12:57:53.749284 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:53.749290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:53.749375 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:53.749523 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:53.750365 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:53.751376 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:53.753120 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:53.753610 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:53.753724 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:53.758167 (+ 4443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:56941: BlockingRecv error: recv got EOF from 127.24.167.2:56941 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 12:57:54 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:54 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:54.447201 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:54.434782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:35947)
0808 12:57:54.435342 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:54.435360 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:54.435473 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:54.435888 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:54.435892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:54.436304 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:54.436506 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:54.436516 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:54.437510 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:54.437514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:54.438702 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:54.438720 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:54.439656 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:54.439662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:54.439848 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:54.443181 (+ 3333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:54.443200 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:54.446844 (+ 3644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":295,"threads_started":1}
W20260808 12:57:54.447371 6474 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:54.434775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:35947 (local address 127.24.167.62:45701)
0808 12:57:54.435373 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:54.435382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:54.435405 (+ 23us) server_negotiation.cc:407] Connection header received
0808 12:57:54.435603 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:54.435608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:54.435699 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:54.435797 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:54.436640 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:54.437376 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:54.438855 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:54.439478 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:54.439616 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:54.447066 (+ 7450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:35947: BlockingRecv error: recv got EOF from 127.24.167.1:35947 (error 108)
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":305,"threads_started":1}
Aug 08 12:57:54 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:54 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:54.690820 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:54.681330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:58809)
0808 12:57:54.681922 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:54.681939 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:54.682059 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:54.682357 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:54.682361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:54.682797 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:54.682970 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:54.682979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:54.684064 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:54.684069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:54.685031 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:54.685043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:54.685739 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:54.685744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:54.685920 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:54.686699 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:54.686718 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:54.690473 (+ 3755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":325,"threads_started":1}
W20260808 12:57:54.690970 6474 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:54.681375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:58809 (local address 127.24.167.62:45701)
0808 12:57:54.681568 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:54.681573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:54.682014 (+ 441us) server_negotiation.cc:407] Connection header received
0808 12:57:54.682157 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:54.682161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:54.682221 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:54.682301 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:54.683115 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:54.683884 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:54.685161 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:54.685579 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:54.685670 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:54.690711 (+ 5041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:58809: BlockingRecv error: recv got EOF from 127.24.167.3:58809 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 08 12:57:54 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:54 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:54.768744 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:54.759286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:44237)
0808 12:57:54.759837 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:54.759854 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:54.759975 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:54.760298 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:54.760301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:54.760752 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:54.760928 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:54.760936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:54.762033 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:54.762037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:54.763053 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:54.763065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:54.763921 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:54.763925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:54.764056 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:54.764834 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:54.764853 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:54.768517 (+ 3664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":312,"threads_started":1}
W20260808 12:57:54.769014 6474 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:54.759348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:44237 (local address 127.24.167.62:45701)
0808 12:57:54.759557 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:54.759562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:54.759936 (+ 374us) server_negotiation.cc:407] Connection header received
0808 12:57:54.760079 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:54.760083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:54.760161 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:54.760250 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:54.761085 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:54.761893 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:54.763205 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:54.763735 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:54.763878 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:54.768715 (+ 4837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:44237: BlockingRecv error: recv got EOF from 127.24.167.2:44237 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 08 12:57:55 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:55 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:55.457902 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:55.448166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:40591)
0808 12:57:55.448805 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:55.448841 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:55.448998 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:55.449321 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:55.449326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:55.449800 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:55.449982 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:55.449991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:55.451162 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:55.451168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:55.452151 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:55.452162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:55.452900 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:55.452907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:55.453080 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:55.453977 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:55.453997 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:55.457530 (+ 3533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":297,"threads_started":1}
W20260808 12:57:55.458065 6481 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:55.448218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:40591 (local address 127.24.167.62:45701)
0808 12:57:55.448821 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:55.448825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:55.448926 (+ 101us) server_negotiation.cc:407] Connection header received
0808 12:57:55.449101 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:55.449104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:55.449164 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:55.449285 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:55.450156 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:55.451018 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:55.452307 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:55.452734 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:55.452866 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:55.457768 (+ 4902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:40591: BlockingRecv error: recv got EOF from 127.24.167.1:40591 (error 108)
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":361,"threads_started":1}
Aug 08 12:57:55 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:55 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:55.701332 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:55.691970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:35695)
0808 12:57:55.692543 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:55.692566 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:55.692737 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:55.693208 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:55.693212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:55.693661 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:55.693901 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:55.693912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:55.694949 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:55.694953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:55.695876 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:55.695887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:55.696556 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:55.696561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:55.696684 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:55.697330 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:55.697348 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:55.701057 (+ 3709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":313,"threads_started":1}
W20260808 12:57:55.701520 6481 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:55.692092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:35695 (local address 127.24.167.62:45701)
0808 12:57:55.692333 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:55.692340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:55.692660 (+ 320us) server_negotiation.cc:407] Connection header received
0808 12:57:55.692918 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:55.692923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:55.693022 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:55.693143 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:55.694051 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:55.694812 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:55.696028 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:55.696426 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:55.696517 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:55.701230 (+ 4713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:35695: BlockingRecv error: recv got EOF from 127.24.167.3:35695 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 12:57:55 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:55 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:55.778684 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:55.769762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:54815)
0808 12:57:55.770401 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:55.770417 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:55.770544 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:55.770863 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:55.770866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:55.771298 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:55.771472 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:55.771480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:55.772469 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:55.772472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:55.773428 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:55.773441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:55.774114 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:55.774118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:55.774232 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:55.774894 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:55.774912 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:55.778355 (+ 3443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":300,"threads_started":1}
W20260808 12:57:55.778915 6481 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:55.769819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:54815 (local address 127.24.167.62:45701)
0808 12:57:55.770100 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:55.770109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:55.770511 (+ 402us) server_negotiation.cc:407] Connection header received
0808 12:57:55.770660 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:55.770663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:55.770732 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:55.770816 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:55.771597 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:55.772335 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:55.773563 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:55.773981 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:55.774077 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:55.778638 (+ 4561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:54815: BlockingRecv error: recv got EOF from 127.24.167.2:54815 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Aug 08 12:57:56 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:56 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:56.468835 6488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:56.459069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:45081)
0808 12:57:56.459652 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:56.459669 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:56.459790 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:56.460129 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:56.460132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:56.460561 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:56.460735 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:56.460744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:56.461832 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:56.461837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:56.462862 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:56.462877 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:56.463598 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:56.463603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:56.463727 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:56.464420 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:56.464438 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:56.468500 (+ 4062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":295,"threads_started":1}
W20260808 12:57:56.468922 6489 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:56.459154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:45081 (local address 127.24.167.62:45701)
0808 12:57:56.459685 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:56.459689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:56.459743 (+ 54us) server_negotiation.cc:407] Connection header received
0808 12:57:56.459891 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:56.459895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:56.459989 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:56.460083 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:56.460878 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:56.461650 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:56.462990 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:56.463414 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:56.463515 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:56.468707 (+ 5192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:45081: BlockingRecv error: recv got EOF from 127.24.167.1:45081 (error 108)
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":305,"threads_started":1}
Aug 08 12:57:56 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:56 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:56.711381 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:56.702311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:39397)
0808 12:57:56.702999 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:56.703016 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:56.703137 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:56.703431 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:56.703434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:56.703848 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:56.704046 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:56.704056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:56.705028 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:56.705031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:56.705973 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:56.705985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:56.706735 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:56.706739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:56.706861 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:56.707449 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:56.707471 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:56.711026 (+ 3555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":325,"threads_started":1}
W20260808 12:57:56.711570 6489 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:56.702543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:39397 (local address 127.24.167.62:45701)
0808 12:57:56.702728 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:56.702733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:56.703092 (+ 359us) server_negotiation.cc:407] Connection header received
0808 12:57:56.703233 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:56.703236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:56.703297 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:56.703381 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:56.704171 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:56.704898 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:56.706095 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:56.706587 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:56.706680 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:56.711270 (+ 4590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:39397: BlockingRecv error: recv got EOF from 127.24.167.3:39397 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 08 12:57:56 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:56 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:56.789283 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:56.779646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:49959)
0808 12:57:56.780265 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:56.780288 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:56.780408 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:56.780733 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:56.780736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:56.781139 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:56.781366 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:56.781375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:56.782684 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:56.782687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:56.783750 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:56.783763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:56.784533 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:56.784540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:56.784713 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:56.785401 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:56.785419 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:56.789038 (+ 3619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":345,"threads_started":1}
W20260808 12:57:56.789683 6489 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:56.779762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:49959 (local address 127.24.167.62:45701)
0808 12:57:56.779935 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:56.779939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:56.780365 (+ 426us) server_negotiation.cc:407] Connection header received
0808 12:57:56.780508 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:56.780511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:56.780586 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:56.780669 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:56.781540 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:56.782543 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:56.783903 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:56.784341 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:56.784498 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:56.789263 (+ 4765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:49959: BlockingRecv error: recv got EOF from 127.24.167.2:49959 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 08 12:57:57 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:57 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:57.480013 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:57.470178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:55511)
0808 12:57:57.470738 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:57.470755 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:57.470859 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:57.471459 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:57.471463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:57.471886 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:57.472072 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:57.472088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:57.473374 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:57.473379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:57.474612 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:57.474625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:57.475297 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:57.475301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:57.475465 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:57.476044 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:57.476062 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:57.479787 (+ 3725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":323,"threads_started":1}
W20260808 12:57:57.480299 6496 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:57.470257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:55511 (local address 127.24.167.62:45701)
0808 12:57:57.471096 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:57.471100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:57.471115 (+ 15us) server_negotiation.cc:407] Connection header received
0808 12:57:57.471165 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:57.471169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:57.471226 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:57.471317 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:57.472237 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:57.473177 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:57.474753 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:57.475168 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:57.475261 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:57.480012 (+ 4751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:55511: BlockingRecv error: recv got EOF from 127.24.167.1:55511 (error 108)
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":595,"threads_started":1}
Aug 08 12:57:57 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:57 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:57.722155 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:57.712384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:60701)
0808 12:57:57.713159 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:57.713175 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:57.713347 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:57.713735 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:57.713740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:57.714309 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:57.714620 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:57.714635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:57.715739 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:57.715745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:57.716695 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:57.716707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:57.717458 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:57.717465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:57.717645 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:57.718221 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:57.718239 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:57.721756 (+ 3517us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":337,"threads_started":1}
W20260808 12:57:57.722177 6496 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:57.712554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:60701 (local address 127.24.167.62:45701)
0808 12:57:57.712727 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:57.712733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:57.713281 (+ 548us) server_negotiation.cc:407] Connection header received
0808 12:57:57.713486 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:57.713492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:57.713601 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:57.713700 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:57.714787 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:57.715603 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:57.716849 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:57.717272 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:57.717408 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:57.721957 (+ 4549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:60701: BlockingRecv error: recv got EOF from 127.24.167.3:60701 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 12:57:57 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:57 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:57.799567 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:57.790252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:33235)
0808 12:57:57.790868 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:57.790884 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:57.791004 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:57.791330 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:57.791334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:57.791750 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:57.791908 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:57.791916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:57.792894 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:57.792897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:57.793856 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:57.793868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:57.794657 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:57.794662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:57.794791 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:57.795456 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:57.795499 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:57.799219 (+ 3720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":356,"threads_started":1}
W20260808 12:57:57.799687 6496 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:57.790416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:33235 (local address 127.24.167.62:45701)
0808 12:57:57.790611 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:57.790615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:57.790965 (+ 350us) server_negotiation.cc:407] Connection header received
0808 12:57:57.791104 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:57.791108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:57.791183 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:57.791266 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:57.792052 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:57.792774 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:57.793999 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:57.794495 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:57.794594 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:57.799419 (+ 4825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:33235: BlockingRecv error: recv got EOF from 127.24.167.2:33235 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 08 12:57:58 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:58 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:58.490875 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:58.480960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:42109)
0808 12:57:58.481570 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:58.481587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:58.481695 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:58.482010 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:58.482014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:58.482459 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:58.482696 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:58.482706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:58.483966 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:58.483970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:58.485232 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:58.485268 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:58.485993 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:58.485998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:58.486118 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:58.486781 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:58.486800 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:58.490476 (+ 3676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":299,"threads_started":1}
W20260808 12:57:58.491226 6503 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:58.481062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:42109 (local address 127.24.167.62:45701)
0808 12:57:58.481639 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:58.481643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:58.481660 (+ 17us) server_negotiation.cc:407] Connection header received
0808 12:57:58.481805 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:58.481809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:58.481869 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:58.481964 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:58.482816 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:58.483794 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:58.485400 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:58.485853 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:58.485950 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:58.490769 (+ 4819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:42109: BlockingRecv error: recv got EOF from 127.24.167.1:42109 (error 108)
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":310,"threads_started":1}
Aug 08 12:57:58 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:58 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:58.732467 6505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:58.723234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:39587)
0808 12:57:58.723843 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:58.723859 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:58.724060 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:58.724439 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:58.724448 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:58.725027 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:58.725208 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:58.725217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:58.726289 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:58.726293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:58.727299 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:58.727311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:58.727964 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:58.727968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:58.728092 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:58.728754 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:58.728773 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:58.732235 (+ 3462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":351,"threads_started":1}
W20260808 12:57:58.732656 6503 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:58.723338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:39587 (local address 127.24.167.62:45701)
0808 12:57:58.723550 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:58.723557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:58.723973 (+ 416us) server_negotiation.cc:407] Connection header received
0808 12:57:58.724186 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:58.724191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:58.724271 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:58.724382 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:58.725315 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:58.726146 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:58.727421 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:58.727811 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:58.727907 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:58.732418 (+ 4511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:39587: BlockingRecv error: recv got EOF from 127.24.167.3:39587 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 12:57:58 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:58 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:58.810671 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:58.800703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:48223)
0808 12:57:58.801269 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:58.801285 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:58.801400 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:58.801717 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:58.801720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:58.802165 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:58.802439 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:58.802488 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:58.803617 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:58.803620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:58.804614 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:58.804626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:58.805308 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:58.805312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:58.805425 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:58.806029 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:58.806046 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:58.810293 (+ 4247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":333,"threads_started":1}
W20260808 12:57:58.810781 6503 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:58.800758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:48223 (local address 127.24.167.62:45701)
0808 12:57:58.800984 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:58.800995 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:58.801362 (+ 367us) server_negotiation.cc:407] Connection header received
0808 12:57:58.801516 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:58.801520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:58.801583 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:58.801666 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:58.802668 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:58.803488 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:58.804764 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:58.805170 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:58.805278 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:58.810522 (+ 5244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:48223: BlockingRecv error: recv got EOF from 127.24.167.2:48223 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 08 12:57:59 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:59 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:59.505048 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:59.491795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:34509)
0808 12:57:59.492383 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:59.492408 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:59.492557 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:59.493087 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:59.493092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:59.493543 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:59.493796 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:59.493806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:59.494998 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:59.495002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:59.495913 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:59.495925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:59.496724 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:59.496729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:59.496925 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:59.501043 (+ 4118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:59.501063 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:59.504821 (+ 3758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":308,"threads_started":1}
W20260808 12:57:59.505393 6512 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:59.491960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:34509 (local address 127.24.167.62:45701)
0808 12:57:59.492775 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:59.492779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:59.492793 (+ 14us) server_negotiation.cc:407] Connection header received
0808 12:57:59.492836 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:59.492840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:59.492895 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:59.493013 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:59.493946 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:59.494843 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:59.496076 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:59.496548 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:59.496692 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:59.505053 (+ 8361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:34509: BlockingRecv error: recv got EOF from 127.24.167.1:34509 (error 108)
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":502,"threads_started":1}
Aug 08 12:57:59 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:59 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:59.742561 6513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:59.733500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:51909)
0808 12:57:59.734079 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:59.734096 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:59.734212 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:59.734548 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:59.734551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:59.734999 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:59.735164 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:59.735174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:59.736226 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:59.736231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:59.737214 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:59.737228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:59.737920 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:59.737924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:59.738069 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:59.738674 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:59.738692 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:59.742264 (+ 3572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":348,"threads_started":1}
W20260808 12:57:59.742944 6512 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:59.733548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:51909 (local address 127.24.167.62:45701)
0808 12:57:59.733755 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:59.733765 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:59.734180 (+ 415us) server_negotiation.cc:407] Connection header received
0808 12:57:59.734313 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:59.734317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:59.734398 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:59.734502 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:59.735298 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:59.736089 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:59.737359 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:59.737783 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:59.737881 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:59.742542 (+ 4661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:51909: BlockingRecv error: recv got EOF from 127.24.167.3:51909 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 08 12:57:59 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:57:59 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:57:59.821170 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:57:59.811833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:33855)
0808 12:57:59.812310 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0808 12:57:59.812326 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:57:59.812446 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:57:59.812766 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:57:59.812769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:57:59.813216 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:57:59.813405 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:59.813413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:59.814461 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:59.814464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:59.815376 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:57:59.815386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:57:59.816039 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:57:59.816043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:57:59.816167 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:59.816771 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:57:59.816788 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:57:59.820935 (+ 4147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":274,"threads_started":1}
W20260808 12:57:59.821411 6512 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:57:59.811911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:33855 (local address 127.24.167.62:45701)
0808 12:57:59.812154 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 12:57:59.812161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:57:59.812410 (+ 249us) server_negotiation.cc:407] Connection header received
0808 12:57:59.812551 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:57:59.812555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:57:59.812634 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:57:59.812717 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:57:59.813538 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:59.814286 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:59.815505 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:57:59.815908 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:57:59.816002 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:57:59.821158 (+ 5156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:33855: BlockingRecv error: recv got EOF from 127.24.167.2:33855 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 08 12:58:00 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:58:00 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:58:00.515828 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:00.506142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:57837)
0808 12:58:00.506776 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:00.506797 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:00.507009 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:00.507446 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:00.507452 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:00.507938 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:00.508113 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:00.508122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:00.509229 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:00.509233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:00.510193 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:00.510205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:00.511216 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:00.511222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:00.511453 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:00.512045 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:58:00.512070 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:00.515467 (+ 3397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":358,"threads_started":1}
W20260808 12:58:00.515928 6518 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:58:00.506224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:57837 (local address 127.24.167.62:45701)
0808 12:58:00.506778 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:00.506785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:00.506872 (+ 87us) server_negotiation.cc:407] Connection header received
0808 12:58:00.507203 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:00.507207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:00.507282 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:00.507404 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:00.508273 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:00.509044 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:00.510349 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:00.511074 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:00.511193 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:00.515671 (+ 4478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:57837: BlockingRecv error: recv got EOF from 127.24.167.1:57837 (error 108)
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":302,"threads_started":1}
Aug 08 12:58:00 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:58:00 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:58:00.753072 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:00.743493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:41139)
0808 12:58:00.744043 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:00.744059 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:00.744231 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:00.744630 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:00.744635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:00.745118 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:00.745307 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:00.745316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:00.746357 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:00.746364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:00.747338 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:00.747351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:00.748174 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:00.748181 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:00.748353 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:00.749054 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:58:00.749078 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:00.752841 (+ 3763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":317,"threads_started":1}
W20260808 12:58:00.753383 6518 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:58:00.743573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:41139 (local address 127.24.167.62:45701)
0808 12:58:00.743800 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:00.743810 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:00.744152 (+ 342us) server_negotiation.cc:407] Connection header received
0808 12:58:00.744386 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:00.744390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:00.744475 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:00.744587 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:00.745470 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:00.746218 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:00.747492 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:00.747965 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:00.748137 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:00.753036 (+ 4899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:41139: BlockingRecv error: recv got EOF from 127.24.167.3:41139 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 12:58:00 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:58:00 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:58:00.831918 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:00.822472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:55159)
0808 12:58:00.823204 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:00.823220 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:00.823337 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:00.823647 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:00.823650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:00.824089 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:00.824264 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:00.824273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:00.825408 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:00.825412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:00.826510 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:00.826523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:00.827240 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:00.827244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:00.827371 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:00.828060 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:58:00.828078 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:00.831561 (+ 3483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":453,"threads_started":1}
W20260808 12:58:00.832049 6518 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:58:00.822493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:55159 (local address 127.24.167.62:45701)
0808 12:58:00.822702 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:00.822711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:00.823313 (+ 602us) server_negotiation.cc:407] Connection header received
0808 12:58:00.823438 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:00.823442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:00.823519 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:00.823603 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:00.824396 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:00.825277 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:00.826642 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:00.827088 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:00.827186 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:00.831788 (+ 4602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:55159: BlockingRecv error: recv got EOF from 127.24.167.2:55159 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 08 12:58:01 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:58:01 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:58:01.527746 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:01.516882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:60533)
0808 12:58:01.517445 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:01.517462 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:01.517580 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:01.517893 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:01.517896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:01.518288 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:01.518505 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:01.518514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:01.519648 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:01.519652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:01.521171 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:01.521187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:01.522545 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:01.522559 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:01.522972 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:01.523602 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:58:01.523621 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:01.527521 (+ 3900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":300,"threads_started":1}
W20260808 12:58:01.528003 6526 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:58:01.516962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:60533 (local address 127.24.167.62:45701)
0808 12:58:01.517475 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:01.517479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:01.517516 (+ 37us) server_negotiation.cc:407] Connection header received
0808 12:58:01.517683 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:01.517687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:01.517747 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:01.517838 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:01.518650 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:01.519509 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:01.521381 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:01.522125 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:01.522252 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:01.527727 (+ 5475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:60533: BlockingRecv error: recv got EOF from 127.24.167.1:60533 (error 108)
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":295,"threads_started":1}
Aug 08 12:58:01 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:58:01 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:58:01.763751 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:01.754053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:34875)
0808 12:58:01.754810 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:01.754826 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:01.754990 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:01.755360 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:01.755365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:01.755836 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:01.756020 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:01.756030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:01.757100 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:01.757106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:01.758064 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:01.758076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:01.758912 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:01.758919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:01.759115 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:01.759719 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:58:01.759739 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:01.763362 (+ 3623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":491,"threads_started":1}
W20260808 12:58:01.763855 6526 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:58:01.754119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:34875 (local address 127.24.167.62:45701)
0808 12:58:01.754291 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:01.754295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:01.754919 (+ 624us) server_negotiation.cc:407] Connection header received
0808 12:58:01.755133 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:01.755138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:01.755213 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:01.755319 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:01.756175 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:01.756935 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:01.758229 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:01.758723 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:01.758861 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:01.763601 (+ 4740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:34875: BlockingRecv error: recv got EOF from 127.24.167.3:34875 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 08 12:58:01 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:58:01 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:58:01.843047 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:01.833067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:33035)
0808 12:58:01.833693 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:01.833710 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:01.833906 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:01.834317 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:01.834322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:01.834843 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:01.835010 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:01.835019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:01.836186 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:01.836194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:01.837240 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:01.837252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:01.838044 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:01.838050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:01.838257 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:01.838910 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:58:01.838929 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:01.842676 (+ 3747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":356,"threads_started":1}
W20260808 12:58:01.843159 6526 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:58:01.833273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:33035 (local address 127.24.167.62:45701)
0808 12:58:01.833571 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:01.833580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:01.833808 (+ 228us) server_negotiation.cc:407] Connection header received
0808 12:58:01.834051 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:01.834057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:01.834158 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:01.834275 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:01.835177 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:01.836028 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:01.837404 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:01.837875 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:01.838018 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:01.842889 (+ 4871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:33035: BlockingRecv error: recv got EOF from 127.24.167.2:33035 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Aug 08 12:58:02 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:58:02 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.1@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:58:02.538846 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:02.529007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.1:51155)
0808 12:58:02.529564 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:02.529581 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:02.529699 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:02.530234 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:02.530236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:02.530732 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:02.530923 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:02.530934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:02.532416 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:02.532421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:02.533438 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:02.533451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:02.534190 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:02.534195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:02.534418 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:02.535096 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:58:02.535114 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:02.538582 (+ 3468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":307,"threads_started":1}
W20260808 12:58:02.539067 6533 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:58:02.529107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:51155 (local address 127.24.167.62:45701)
0808 12:58:02.529956 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:02.529961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:02.529977 (+ 16us) server_negotiation.cc:407] Connection header received
0808 12:58:02.530031 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:02.530035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:02.530093 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:02.530188 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:02.531082 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:02.532275 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:02.533582 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:02.534034 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:02.534163 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:02.538813 (+ 4650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.1:51155: BlockingRecv error: recv got EOF from 127.24.167.1:51155 (error 108)
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":607,"threads_started":1}
Aug 08 12:58:02 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:58:02 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.3@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:58:02.776723 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:02.764974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.3:44087)
0808 12:58:02.765492 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:02.765508 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:02.765637 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:02.766031 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:02.766039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:02.766797 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:02.767059 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:02.767072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:02.768277 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:02.768297 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:02.770167 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:02.770187 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:02.771482 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:02.771488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:02.771687 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:02.772329 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:58:02.772349 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:02.776481 (+ 4132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":264,"threads_started":1}
W20260808 12:58:02.776986 6533 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:58:02.765057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:44087 (local address 127.24.167.62:45701)
0808 12:58:02.765369 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:02.765375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:02.765639 (+ 264us) server_negotiation.cc:407] Connection header received
0808 12:58:02.765738 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:02.765742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:02.765809 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:02.765899 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:02.767279 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:02.768078 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:02.770569 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:02.771288 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:02.771436 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:02.776738 (+ 5302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.3:44087: BlockingRecv error: recv got EOF from 127.24.167.3:44087 (error 108)
Metrics: {"server-negotiator.queue_time_us":165}
Aug 08 12:58:02 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 12:58:02 dist-test-slave-tk9z krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.167.2@KRBTEST.COM for oryx/127.24.167.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 12:58:02.854476 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:02.844502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45701 (local address 127.24.167.2:51629)
0808 12:58:02.845029 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:02.845048 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:02.845169 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:02.845534 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:02.845538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:02.846065 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:02.846328 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:02.846343 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:02.847760 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:02.847764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:02.848832 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:02.848852 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:02.849536 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:02.849540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:02.849682 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:02.850317 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:58:02.850335 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:02.854064 (+ 3729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.167.62:45701: Server oryx/127.24.167.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":282,"threads_started":1}
W20260808 12:58:02.854823 6533 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:58:02.844715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:51629 (local address 127.24.167.62:45701)
0808 12:58:02.845000 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:02.845008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:02.845153 (+ 145us) server_negotiation.cc:407] Connection header received
0808 12:58:02.845269 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:02.845273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:02.845340 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:02.845505 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:02.846543 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:02.847633 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:02.848992 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:02.849391 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:02.849486 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:02.854313 (+ 4827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.167.2:51629: BlockingRecv error: recv got EOF from 127.24.167.2:51629 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
I20260808 12:58:03.386777 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 5790
I20260808 12:58:03.399580 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:03.387917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40118 (local address 127.24.167.1:33099)
0808 12:58:03.388568 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:03.388575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:03.388602 (+ 27us) server_negotiation.cc:407] Connection header received
0808 12:58:03.388677 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:03.388683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:03.388917 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:03.389115 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:03.390583 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:03.391665 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:03.392861 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:03.393435 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:03.393545 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:03.396173 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:03.396182 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:03.396197 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:58:03.396247 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:03.399065 (+ 2818us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.167.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 12:58:03.399369 (+ 304us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40118: Request ticket server kudu/127.24.167.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":366,"threads_started":1}
W20260808 12:58:03.399785 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 5790: Not authorized: Client connection negotiation failed: client connection to 127.24.167.1:33099: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.167.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 12:58:03.400010 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 5790
I20260808 12:58:03.429950 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 6025
W20260808 12:58:03.442819 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 6025: Not authorized: Client connection negotiation failed: client connection to 127.24.167.2:38601: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.167.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 12:58:03.442895 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:03.431144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.24.167.2:38601)
0808 12:58:03.431765 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:03.431772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:03.431796 (+ 24us) server_negotiation.cc:407] Connection header received
0808 12:58:03.431866 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:03.431873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:03.432099 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:03.432306 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:03.433656 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:03.434823 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:03.436108 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:03.436660 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:03.436794 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:03.439417 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:03.439431 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:03.439446 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:58:03.439494 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:03.442052 (+ 2558us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.167.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 12:58:03.442576 (+ 524us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54762: Request ticket server kudu/127.24.167.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":331,"threads_started":1}
W20260808 12:58:03.442988 6540 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54762: Request ticket server kudu/127.24.167.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 12:58:03.443027 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 6025
I20260808 12:58:03.467912 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 6225
I20260808 12:58:03.481295 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:03.469119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.24.167.3:38809)
0808 12:58:03.469677 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:03.469684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:03.469709 (+ 25us) server_negotiation.cc:407] Connection header received
0808 12:58:03.469773 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:03.469780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:03.470030 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:03.470192 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:03.472235 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:03.473267 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:03.474528 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:03.475101 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:03.475206 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:03.477746 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:03.477756 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:03.477772 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:58:03.477827 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:03.480693 (+ 2866us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.167.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 12:58:03.480989 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60634: Request ticket server kudu/127.24.167.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":252,"threads_started":1}
W20260808 12:58:03.481410 6542 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60634: Request ticket server kudu/127.24.167.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 12:58:03.481389 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 6225: Not authorized: Client connection negotiation failed: client connection to 127.24.167.3:38809: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.167.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 12:58:03.481580 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 6225
I20260808 12:58:03.505335 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 5696
I20260808 12:58:03.506840 5782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:58:03.628676 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 5696
2026-08-08T12:58:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22005 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T12:58:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:58:03Z Disabled control of system clock
I20260808 12:58:03.724710 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:43891
--webserver_interface=127.24.167.62
--webserver_port=0
--builtin_ntp_servers=127.24.167.20:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:43891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 12:58:04.076352 6548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:58:04.076673 6548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:58:04.076813 6548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:58:04.084395 6548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:58:04.084520 6548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:58:04.084619 6548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:58:04.084724 6548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:58:04.084816 6548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:58:04.094390 6548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:43891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:43891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:58:04.095831 6548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:58:04.097664 6548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:58:04.110095 6561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:04.110251 6554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:04.110210 6556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:58:04.110857 6548 server_base.cc:1034] running on GCE node
I20260808 12:58:04.111665 6548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:58:04.113078 6548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:58:04.114285 6548 hybrid_clock.cc:648] HybridClock initialized: now 1786193884114235 us; error 52 us; skew 500 ppm
I20260808 12:58:04.114678 6548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:58:04.117105 6548 webserver.cc:460] Webserver started at http://127.24.167.62:34475/ using document root <none> and password file <none>
I20260808 12:58:04.117762 6548 fs_manager.cc:359] Metadata directory not provided
I20260808 12:58:04.117853 6548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:58:04.118135 6548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:58:04.121203 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "8c258934a3454116ab20d758cd9d6186"
format_stamp: "Formatted at 2026-08-08 12:58:04 on dist-test-slave-tk9z"
server_key: "05fcfaf07b30c27d433e32213b0d703f"
server_key_iv: "9fd9d4b07ed9ce82c35c70159c262894"
server_key_version: "encryptionkey@0"
I20260808 12:58:04.121937 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "8c258934a3454116ab20d758cd9d6186"
format_stamp: "Formatted at 2026-08-08 12:58:04 on dist-test-slave-tk9z"
server_key: "05fcfaf07b30c27d433e32213b0d703f"
server_key_iv: "9fd9d4b07ed9ce82c35c70159c262894"
server_key_version: "encryptionkey@0"
I20260808 12:58:04.127135 6548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 12:58:04.130537 6575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:04.131855 6548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 12:58:04.132035 6548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "8c258934a3454116ab20d758cd9d6186"
format_stamp: "Formatted at 2026-08-08 12:58:04 on dist-test-slave-tk9z"
server_key: "05fcfaf07b30c27d433e32213b0d703f"
server_key_iv: "9fd9d4b07ed9ce82c35c70159c262894"
server_key_version: "encryptionkey@0"
I20260808 12:58:04.132272 6548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:58:04.153393 6548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:58:04.154331 6548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:58:04.154683 6548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:58:04.174136 6548 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:43891
I20260808 12:58:04.174153 6666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:43891 every 8 connection(s)
I20260808 12:58:04.175696 6548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:58:04.179713 6668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:58:04.181843 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 6548
I20260808 12:58:04.182099 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:58:04.182839 25244 external_mini_cluster.cc:1428] Setting key 2fd6d0da511ae8576914180b11275a15
I20260808 12:58:04.192003 6668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186: Bootstrap starting.
I20260808 12:58:04.195861 6668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186: Neither blocks nor log segments found. Creating new log.
I20260808 12:58:04.197049 6668 log.cc:824] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186: Log is configured to *not* fsync() on all Append() calls
I20260808 12:58:04.197495 6675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:04.185102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.24.167.62:43891)
0808 12:58:04.186121 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:04.186132 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:04.186175 (+ 43us) server_negotiation.cc:407] Connection header received
0808 12:58:04.187914 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:04.187938 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:04.188421 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:04.188752 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:04.189537 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:04.190922 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:04.192863 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:04.193532 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:04.193686 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:04.195040 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:04.195047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:04.195060 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:58:04.195614 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:04.195705 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:58:04.195729 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:04.195835 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:04.196158 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 12:58:04.196518 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":380,"threads_started":1}
I20260808 12:58:04.200184 6668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186: No bootstrap required, opened a new log
I20260808 12:58:04.207505 6668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c258934a3454116ab20d758cd9d6186" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43891 } }
I20260808 12:58:04.207904 6668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:58:04.208027 6668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c258934a3454116ab20d758cd9d6186, State: Initialized, Role: FOLLOWER
I20260808 12:58:04.208611 6668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c258934a3454116ab20d758cd9d6186" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43891 } }
I20260808 12:58:04.209021 6668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:58:04.209208 6668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:58:04.209455 6668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:58:04.211874 6668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c258934a3454116ab20d758cd9d6186" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43891 } }
I20260808 12:58:04.212422 6668 leader_election.cc:302] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c258934a3454116ab20d758cd9d6186; no voters:
I20260808 12:58:04.213088 6668 leader_election.cc:288] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:58:04.213346 6678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:58:04.214165 6678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [term 1 LEADER]: Becoming Leader. State: Replica: 8c258934a3454116ab20d758cd9d6186, State: Running, Role: LEADER
I20260808 12:58:04.214769 6678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c258934a3454116ab20d758cd9d6186" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43891 } }
I20260808 12:58:04.215449 6668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:58:04.217851 6679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c258934a3454116ab20d758cd9d6186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c258934a3454116ab20d758cd9d6186" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43891 } } }
I20260808 12:58:04.218125 6680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c258934a3454116ab20d758cd9d6186. Latest consensus state: current_term: 1 leader_uuid: "8c258934a3454116ab20d758cd9d6186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c258934a3454116ab20d758cd9d6186" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43891 } } }
I20260808 12:58:04.218564 6679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [sys.catalog]: This master's current role is: LEADER
I20260808 12:58:04.218593 6680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186 [sys.catalog]: This master's current role is: LEADER
I20260808 12:58:04.219815 6692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:58:04.226632 6692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:58:04.237282 6692 catalog_manager.cc:1347] Generated new cluster ID: b9401a03e78b4e79ae3680c69b785b3d
I20260808 12:58:04.237437 6692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:58:04.272019 6692 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:58:04.273352 6692 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:58:04.294030 6692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c258934a3454116ab20d758cd9d6186: Generated new TSK 0
I20260808 12:58:04.295270 6692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 12:58:04.311172 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:43891
--builtin_ntp_servers=127.24.167.20:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 12:58:04.660874 6706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:58:04.661170 6706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:58:04.661337 6706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:58:04.668782 6706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:58:04.668900 6706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:58:04.669075 6706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:58:04.678614 6706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:43891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:58:04.680045 6706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:58:04.681883 6706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:58:04.695367 6714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:04.695297 6718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:04.695341 6715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:58:04.696713 6706 server_base.cc:1034] running on GCE node
I20260808 12:58:04.697379 6706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:58:04.698527 6706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:58:04.699782 6706 hybrid_clock.cc:648] HybridClock initialized: now 1786193884699728 us; error 52 us; skew 500 ppm
I20260808 12:58:04.700140 6706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:58:04.706954 6706 webserver.cc:460] Webserver started at http://127.24.167.1:32885/ using document root <none> and password file <none>
I20260808 12:58:04.707721 6706 fs_manager.cc:359] Metadata directory not provided
I20260808 12:58:04.707862 6706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:58:04.708266 6706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:58:04.711407 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "2b1e9bff708f407bbb79ec89b86589ae"
format_stamp: "Formatted at 2026-08-08 12:58:04 on dist-test-slave-tk9z"
server_key: "a18c5a6aa5c542f246f38650b2958827"
server_key_iv: "0e55a7af51f058d3dca6aeceb1d8eab5"
server_key_version: "encryptionkey@0"
I20260808 12:58:04.712266 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "2b1e9bff708f407bbb79ec89b86589ae"
format_stamp: "Formatted at 2026-08-08 12:58:04 on dist-test-slave-tk9z"
server_key: "a18c5a6aa5c542f246f38650b2958827"
server_key_iv: "0e55a7af51f058d3dca6aeceb1d8eab5"
server_key_version: "encryptionkey@0"
I20260808 12:58:04.718065 6706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 12:58:04.721875 6724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:04.723260 6706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:58:04.723667 6706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "2b1e9bff708f407bbb79ec89b86589ae"
format_stamp: "Formatted at 2026-08-08 12:58:04 on dist-test-slave-tk9z"
server_key: "a18c5a6aa5c542f246f38650b2958827"
server_key_iv: "0e55a7af51f058d3dca6aeceb1d8eab5"
server_key_version: "encryptionkey@0"
I20260808 12:58:04.723958 6706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:58:04.739260 6706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:58:04.740126 6706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:58:04.740444 6706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:58:04.741427 6706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:58:04.743001 6706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:58:04.743127 6706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:04.743270 6706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:58:04.743364 6706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:04.777215 6706 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:45869
I20260808 12:58:04.777271 6906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:45869 every 8 connection(s)
I20260808 12:58:04.778606 6706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:58:04.780071 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 6706
I20260808 12:58:04.780295 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:58:04.781164 25244 external_mini_cluster.cc:1428] Setting key 8ba670408fef68d86cd9ac7a98bfa20d
I20260808 12:58:04.784072 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:43891
--builtin_ntp_servers=127.24.167.20:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 12:58:04.792199 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:04.781333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:55581 (local address 127.24.167.62:43891)
0808 12:58:04.782134 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:04.782140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:04.783273 (+ 1133us) server_negotiation.cc:407] Connection header received
0808 12:58:04.785136 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:04.785142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:04.785231 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:04.785351 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:04.786416 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:04.787176 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:04.788735 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:04.789427 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:04.789620 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:04.791218 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:04.791225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:04.791231 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:58:04.791261 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:04.791307 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:58:04.791319 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:04.791434 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:04.791694 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 12:58:04.791806 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":539,"threads_started":1}
I20260808 12:58:04.793355 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:04.782002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43891 (local address 127.24.167.1:55581)
0808 12:58:04.783119 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:04.783170 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:04.784869 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:04.785598 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:04.785605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:04.785687 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:04.786232 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:04.786249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:04.787330 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:04.787335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:04.788586 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:04.788599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:04.789717 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:04.789725 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:04.789989 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:04.790851 (+ 862us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 12:58:04.790878 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:04.790922 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 12:58:04.791006 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:58:04.791450 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:58:04.791463 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:58:04.791477 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 12:58:04.791679 (+ 202us) client_negotiation.cc:240] Negotiation successful
0808 12:58:04.792122 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":387,"threads_started":1}
I20260808 12:58:04.795185 6907 heartbeater.cc:347] Connected to a master server at 127.24.167.62:43891
I20260808 12:58:04.795575 6907 heartbeater.cc:464] Registering TS with master...
I20260808 12:58:04.796624 6907 heartbeater.cc:511] Master 127.24.167.62:43891 requested a full tablet report, sending...
I20260808 12:58:04.798993 6602 ts_manager.cc:194] Registered new tserver with Master: 2b1e9bff708f407bbb79ec89b86589ae (127.24.167.1:45869)
I20260808 12:58:04.800427 6602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.167.1:55581
W20260808 12:58:05.160729 6912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:58:05.161006 6912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:58:05.161141 6912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:58:05.168550 6912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:58:05.168664 6912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:58:05.168833 6912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:58:05.178754 6912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:43891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:58:05.180111 6912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:58:05.182029 6912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:58:05.195931 6918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:05.196018 6920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:05.196866 6922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:58:05.197149 6912 server_base.cc:1034] running on GCE node
I20260808 12:58:05.197858 6912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:58:05.198958 6912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:58:05.200161 6912 hybrid_clock.cc:648] HybridClock initialized: now 1786193885200097 us; error 54 us; skew 500 ppm
I20260808 12:58:05.200502 6912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:58:05.203321 6912 webserver.cc:460] Webserver started at http://127.24.167.2:46865/ using document root <none> and password file <none>
I20260808 12:58:05.204037 6912 fs_manager.cc:359] Metadata directory not provided
I20260808 12:58:05.204156 6912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:58:05.204506 6912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:58:05.207448 6912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "d57a20691ef945baa102d51bc023fdba"
format_stamp: "Formatted at 2026-08-08 12:58:05 on dist-test-slave-tk9z"
server_key: "eb475e6836b0d02651680f63757dba7b"
server_key_iv: "308421455b3704488405f4623072959a"
server_key_version: "encryptionkey@0"
I20260808 12:58:05.208253 6912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "d57a20691ef945baa102d51bc023fdba"
format_stamp: "Formatted at 2026-08-08 12:58:05 on dist-test-slave-tk9z"
server_key: "eb475e6836b0d02651680f63757dba7b"
server_key_iv: "308421455b3704488405f4623072959a"
server_key_version: "encryptionkey@0"
I20260808 12:58:05.213464 6912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 12:58:05.217187 6933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:05.218797 6912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:58:05.218992 6912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "d57a20691ef945baa102d51bc023fdba"
format_stamp: "Formatted at 2026-08-08 12:58:05 on dist-test-slave-tk9z"
server_key: "eb475e6836b0d02651680f63757dba7b"
server_key_iv: "308421455b3704488405f4623072959a"
server_key_version: "encryptionkey@0"
I20260808 12:58:05.219215 6912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:58:05.245863 6912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:58:05.246788 6912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:58:05.247120 6912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:58:05.248171 6912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:58:05.249701 6912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:58:05.249822 6912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:05.249967 6912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:58:05.250062 6912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:05.284564 6912 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:43049
I20260808 12:58:05.284775 7125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:43049 every 8 connection(s)
I20260808 12:58:05.285931 6912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:58:05.286528 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 6912
I20260808 12:58:05.286772 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:58:05.287420 25244 external_mini_cluster.cc:1428] Setting key c16d74421c9afa0c7b4225495f579051
I20260808 12:58:05.290934 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:43891
--builtin_ntp_servers=127.24.167.20:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 12:58:05.299587 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:05.289166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:39903 (local address 127.24.167.62:43891)
0808 12:58:05.289373 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:05.289381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:05.291090 (+ 1709us) server_negotiation.cc:407] Connection header received
0808 12:58:05.292932 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:05.292940 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:05.293035 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:05.293148 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:05.294255 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:05.295191 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:05.296777 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:05.297174 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:05.297307 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:05.298762 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:05.298766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:05.298770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:58:05.298788 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:05.298820 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:58:05.298831 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:05.298925 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:05.299170 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 12:58:05.299267 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 12:58:05.300602 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:05.289802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43891 (local address 127.24.167.2:39903)
0808 12:58:05.290922 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:05.290969 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:05.292665 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:05.293373 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:05.293381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:05.293479 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:05.294044 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:05.294061 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:05.295354 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:05.295365 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:05.296615 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:05.296627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:05.297413 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:05.297422 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:05.297643 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:05.298411 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 12:58:05.298417 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:05.298489 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 12:58:05.298570 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:58:05.298937 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:58:05.298949 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:58:05.298958 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 12:58:05.299142 (+ 184us) client_negotiation.cc:240] Negotiation successful
0808 12:58:05.299541 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":335,"threads_started":1}
I20260808 12:58:05.302096 7128 heartbeater.cc:347] Connected to a master server at 127.24.167.62:43891
I20260808 12:58:05.302464 7128 heartbeater.cc:464] Registering TS with master...
I20260808 12:58:05.303232 7128 heartbeater.cc:511] Master 127.24.167.62:43891 requested a full tablet report, sending...
I20260808 12:58:05.304697 6602 ts_manager.cc:194] Registered new tserver with Master: d57a20691ef945baa102d51bc023fdba (127.24.167.2:43049)
I20260808 12:58:05.305498 6602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.167.2:39903
W20260808 12:58:05.637415 7134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:58:05.637753 7134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:58:05.637926 7134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:58:05.645120 7134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:58:05.645279 7134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:58:05.645473 7134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:58:05.654868 7134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:43891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:58:05.656239 7134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:58:05.658061 7134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:58:05.671442 7140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:05.671393 7143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:05.671607 7141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:58:05.672499 7134 server_base.cc:1034] running on GCE node
I20260808 12:58:05.673182 7134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:58:05.674345 7134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:58:05.675648 7134 hybrid_clock.cc:648] HybridClock initialized: now 1786193885675575 us; error 71 us; skew 500 ppm
I20260808 12:58:05.676028 7134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:58:05.678648 7134 webserver.cc:460] Webserver started at http://127.24.167.3:36231/ using document root <none> and password file <none>
I20260808 12:58:05.679432 7134 fs_manager.cc:359] Metadata directory not provided
I20260808 12:58:05.679584 7134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:58:05.679965 7134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:58:05.683220 7134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "e10617912e864eb0925a3504a1c0e8be"
format_stamp: "Formatted at 2026-08-08 12:58:05 on dist-test-slave-tk9z"
server_key: "084cdafa727a57535967ff68e1ea6469"
server_key_iv: "74ff29699817cb0a070d5ce71d62d2c5"
server_key_version: "encryptionkey@0"
I20260808 12:58:05.684207 7134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "e10617912e864eb0925a3504a1c0e8be"
format_stamp: "Formatted at 2026-08-08 12:58:05 on dist-test-slave-tk9z"
server_key: "084cdafa727a57535967ff68e1ea6469"
server_key_iv: "74ff29699817cb0a070d5ce71d62d2c5"
server_key_version: "encryptionkey@0"
I20260808 12:58:05.689790 7134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 12:58:05.693569 7154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:05.694914 7134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:58:05.695124 7134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "e10617912e864eb0925a3504a1c0e8be"
format_stamp: "Formatted at 2026-08-08 12:58:05 on dist-test-slave-tk9z"
server_key: "084cdafa727a57535967ff68e1ea6469"
server_key_iv: "74ff29699817cb0a070d5ce71d62d2c5"
server_key_version: "encryptionkey@0"
I20260808 12:58:05.695353 7134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:58:05.709758 7134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:58:05.710677 7134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:58:05.710996 7134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:58:05.712006 7134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:58:05.713624 7134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:58:05.713716 7134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:05.713794 7134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:58:05.713851 7134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:05.749486 7134 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:45917
I20260808 12:58:05.749586 7331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:45917 every 8 connection(s)
I20260808 12:58:05.750914 7134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:58:05.760331 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 7134
I20260808 12:58:05.760552 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:58:05.761075 25244 external_mini_cluster.cc:1428] Setting key 2266f0d058507d79734dd542cbc04e43
I20260808 12:58:05.763773 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:05.753529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:46635 (local address 127.24.167.62:43891)
0808 12:58:05.753715 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:05.753720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:05.755154 (+ 1434us) server_negotiation.cc:407] Connection header received
0808 12:58:05.756942 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:05.756956 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:05.757048 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:05.757160 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:05.758228 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:05.759048 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:05.760641 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:05.761071 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:05.761212 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:05.762855 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:05.762860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:05.762863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:58:05.762881 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:05.762914 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:58:05.762926 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:05.763027 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:05.763266 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 12:58:05.763366 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 12:58:05.764851 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:05.754036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43891 (local address 127.24.167.3:46635)
0808 12:58:05.754996 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:05.755032 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:05.756671 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:05.757407 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:05.757415 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:05.757505 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:05.758060 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:05.758076 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:05.759201 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:05.759210 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:05.760458 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:05.760496 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:05.761305 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:05.761313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:05.761585 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:05.762480 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 12:58:05.762484 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:05.762544 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 12:58:05.762644 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:58:05.763025 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:58:05.763038 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:58:05.763047 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 12:58:05.763251 (+ 204us) client_negotiation.cc:240] Negotiation successful
0808 12:58:05.763649 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":361,"threads_started":1}
I20260808 12:58:05.766243 7334 heartbeater.cc:347] Connected to a master server at 127.24.167.62:43891
I20260808 12:58:05.766625 7334 heartbeater.cc:464] Registering TS with master...
I20260808 12:58:05.767478 7334 heartbeater.cc:511] Master 127.24.167.62:43891 requested a full tablet report, sending...
I20260808 12:58:05.769034 6602 ts_manager.cc:194] Registered new tserver with Master: e10617912e864eb0925a3504a1c0e8be (127.24.167.3:45917)
I20260808 12:58:05.769955 6602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.167.3:46635
I20260808 12:58:05.777127 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 12:58:05.784756 6909 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:58:05.783107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49116 (local address 127.24.167.62:43891)
0808 12:58:05.783297 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:05.783304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:05.783875 (+ 571us) server_negotiation.cc:407] Connection header received
0808 12:58:05.783986 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:05.783989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:05.784053 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:05.784155 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:05.784572 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49116: BlockingRecv error: recv got EOF from 127.0.0.1:49116 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 12:58:05.786044 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 6706
I20260808 12:58:05.794162 7347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:05.787052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.24.167.1:45869)
0808 12:58:05.787664 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:05.787670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:05.787696 (+ 26us) server_negotiation.cc:407] Connection header received
0808 12:58:05.787748 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:05.787754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:05.787981 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:05.788106 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:05.788520 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:05.789525 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:05.790899 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:05.791470 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:05.791574 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:05.792745 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:05.792754 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:05.792767 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:58:05.793339 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:05.793388 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:58:05.793407 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:05.793513 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:05.793629 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 12:58:05.793746 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":322,"threads_started":1}
I20260808 12:58:05.795470 6898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:58:05.878592 6907 heartbeater.cc:503] Master 127.24.167.62:43891 was elected leader, sending a full tablet report...
I20260808 12:58:05.879887 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 6706
W20260808 12:58:05.903779 6577 connection.cc:535] server connection from 127.24.167.1:55581 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 12:58:05.904145 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 6912
I20260808 12:58:05.912467 7350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:05.905082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59274 (local address 127.24.167.2:43049)
0808 12:58:05.905618 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:05.905626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:05.905653 (+ 27us) server_negotiation.cc:407] Connection header received
0808 12:58:05.905719 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:05.905725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:05.905958 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:05.906151 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:05.906642 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:05.907686 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:05.908952 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:05.909524 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:05.909682 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:05.910983 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:05.910991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:05.911006 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:58:05.911587 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:05.911633 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:58:05.911652 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:05.911784 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:05.911961 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 12:58:05.912093 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":261,"threads_started":1}
I20260808 12:58:05.913739 7118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:58:05.995168 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 6912
I20260808 12:58:06.017714 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 7134
I20260808 12:58:06.026335 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:06.018689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.24.167.3:45917)
0808 12:58:06.019282 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:06.019297 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:06.019326 (+ 29us) server_negotiation.cc:407] Connection header received
0808 12:58:06.019391 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:06.019398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:06.019615 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:06.019790 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:06.020308 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:06.021456 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:06.022985 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:06.023550 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:06.023701 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:06.024800 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:06.024809 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:06.024824 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:58:06.025431 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:06.025488 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:58:06.025508 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:06.025651 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:06.025807 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 12:58:06.025938 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":288,"threads_started":1}
I20260808 12:58:06.027660 7318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:58:06.110257 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 7134
I20260808 12:58:06.136512 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 6548
I20260808 12:58:06.137949 6662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:58:06.242712 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 6548
2026-08-08T12:58:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2608 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T12:58:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:58:06Z Disabled control of system clock
I20260808 12:58:06.331655 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:35431
--webserver_interface=127.24.167.62
--webserver_port=0
--builtin_ntp_servers=127.24.167.20:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:35431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 12:58:06.691215 7363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:58:06.691526 7363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:58:06.691684 7363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:58:06.699163 7363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:58:06.699271 7363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:58:06.699417 7363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:58:06.699520 7363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:58:06.699605 7363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:58:06.709023 7363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:35431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:35431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:58:06.710420 7363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:58:06.712332 7363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:58:06.724711 7369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:06.724666 7370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:06.724752 7374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:58:06.725768 7363 server_base.cc:1034] running on GCE node
I20260808 12:58:06.726624 7363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:58:06.728040 7363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:58:06.729285 7363 hybrid_clock.cc:648] HybridClock initialized: now 1786193886729251 us; error 46 us; skew 500 ppm
I20260808 12:58:06.729655 7363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:58:06.732023 7363 webserver.cc:460] Webserver started at http://127.24.167.62:43919/ using document root <none> and password file <none>
I20260808 12:58:06.732738 7363 fs_manager.cc:359] Metadata directory not provided
I20260808 12:58:06.732873 7363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:58:06.733209 7363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:58:06.736253 7363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "1d5a4a1ff36745dd947ebc837640f428"
format_stamp: "Formatted at 2026-08-08 12:58:06 on dist-test-slave-tk9z"
server_key: "90d4c10e559d4812655ef0e8b18afcd1"
server_key_iv: "21533cf8b6a3ea9fd4a8150ab8ac6dad"
server_key_version: "encryptionkey@0"
I20260808 12:58:06.737118 7363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "1d5a4a1ff36745dd947ebc837640f428"
format_stamp: "Formatted at 2026-08-08 12:58:06 on dist-test-slave-tk9z"
server_key: "90d4c10e559d4812655ef0e8b18afcd1"
server_key_iv: "21533cf8b6a3ea9fd4a8150ab8ac6dad"
server_key_version: "encryptionkey@0"
I20260808 12:58:06.742281 7363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 12:58:06.745774 7386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:06.747274 7363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 12:58:06.747512 7363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "1d5a4a1ff36745dd947ebc837640f428"
format_stamp: "Formatted at 2026-08-08 12:58:06 on dist-test-slave-tk9z"
server_key: "90d4c10e559d4812655ef0e8b18afcd1"
server_key_iv: "21533cf8b6a3ea9fd4a8150ab8ac6dad"
server_key_version: "encryptionkey@0"
I20260808 12:58:06.747751 7363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:58:06.753628 7363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:58:06.754544 7363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:58:06.754851 7363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:58:06.775180 7363 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:35431
I20260808 12:58:06.775228 7468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:35431 every 8 connection(s)
I20260808 12:58:06.776764 7363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:58:06.778030 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 7363
I20260808 12:58:06.778278 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:58:06.779052 25244 external_mini_cluster.cc:1428] Setting key bafeeb247fb762384f74dac29ba0d6fb
I20260808 12:58:06.781495 7469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:58:06.788986 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:06.781387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.24.167.62:35431)
0808 12:58:06.782585 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:06.782600 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:06.782653 (+ 53us) server_negotiation.cc:407] Connection header received
0808 12:58:06.784770 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:06.784798 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:06.785344 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:06.785803 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:06.786891 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:06.786899 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:06.786914 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:58:06.786932 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:06.787026 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:58:06.787049 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:06.787282 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:06.787430 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 12:58:06.787713 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":411,"threads_started":1}
I20260808 12:58:06.795728 7469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428: Bootstrap starting.
I20260808 12:58:06.799015 7469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428: Neither blocks nor log segments found. Creating new log.
I20260808 12:58:06.800236 7469 log.cc:824] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428: Log is configured to *not* fsync() on all Append() calls
I20260808 12:58:06.803256 7469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428: No bootstrap required, opened a new log
I20260808 12:58:06.811475 7469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5a4a1ff36745dd947ebc837640f428" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35431 } }
I20260808 12:58:06.811934 7469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:58:06.812063 7469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5a4a1ff36745dd947ebc837640f428, State: Initialized, Role: FOLLOWER
I20260808 12:58:06.812714 7469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5a4a1ff36745dd947ebc837640f428" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35431 } }
I20260808 12:58:06.813000 7469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:58:06.813156 7469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:58:06.813344 7469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:58:06.815940 7469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5a4a1ff36745dd947ebc837640f428" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35431 } }
I20260808 12:58:06.816542 7469 leader_election.cc:302] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d5a4a1ff36745dd947ebc837640f428; no voters:
I20260808 12:58:06.817260 7469 leader_election.cc:288] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:58:06.817408 7477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:58:06.818750 7477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [term 1 LEADER]: Becoming Leader. State: Replica: 1d5a4a1ff36745dd947ebc837640f428, State: Running, Role: LEADER
I20260808 12:58:06.819305 7477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5a4a1ff36745dd947ebc837640f428" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35431 } }
I20260808 12:58:06.819684 7469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:58:06.822311 7480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d5a4a1ff36745dd947ebc837640f428. Latest consensus state: current_term: 1 leader_uuid: "1d5a4a1ff36745dd947ebc837640f428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5a4a1ff36745dd947ebc837640f428" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35431 } } }
I20260808 12:58:06.822803 7480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [sys.catalog]: This master's current role is: LEADER
I20260808 12:58:06.822664 7478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d5a4a1ff36745dd947ebc837640f428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5a4a1ff36745dd947ebc837640f428" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35431 } } }
I20260808 12:58:06.822994 7478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428 [sys.catalog]: This master's current role is: LEADER
I20260808 12:58:06.824872 7489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:58:06.829528 7489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:58:06.838205 7489 catalog_manager.cc:1347] Generated new cluster ID: 3677cf7e59c64c699f5f817732329028
I20260808 12:58:06.838321 7489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:58:06.861935 7489 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:58:06.862483 7489 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:58:06.874922 7489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d5a4a1ff36745dd947ebc837640f428: Generated new TSK 0
I20260808 12:58:06.875864 7489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 12:58:06.886894 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35431
--builtin_ntp_servers=127.24.167.20:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 12:58:07.262267 7504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:58:07.262651 7504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:58:07.262761 7504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:58:07.270169 7504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:58:07.270283 7504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:58:07.270473 7504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:58:07.279742 7504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:58:07.281006 7504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:58:07.282840 7504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:58:07.297223 7514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:07.297329 7522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:07.297318 7515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:58:07.298197 7504 server_base.cc:1034] running on GCE node
I20260808 12:58:07.298960 7504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:58:07.300100 7504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:58:07.301375 7504 hybrid_clock.cc:648] HybridClock initialized: now 1786193887301318 us; error 59 us; skew 500 ppm
I20260808 12:58:07.301750 7504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:58:07.304127 7504 webserver.cc:460] Webserver started at http://127.24.167.1:44035/ using document root <none> and password file <none>
I20260808 12:58:07.304875 7504 fs_manager.cc:359] Metadata directory not provided
I20260808 12:58:07.305023 7504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:58:07.305369 7504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:58:07.308369 7504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "03a4137e9f9648e696d2f24066f5fe95"
format_stamp: "Formatted at 2026-08-08 12:58:07 on dist-test-slave-tk9z"
server_key: "49d430ac54c51b5ffb449b0554a94061"
server_key_iv: "d0a8746c68b319d8ca9320f3957f810e"
server_key_version: "encryptionkey@0"
I20260808 12:58:07.309208 7504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "03a4137e9f9648e696d2f24066f5fe95"
format_stamp: "Formatted at 2026-08-08 12:58:07 on dist-test-slave-tk9z"
server_key: "49d430ac54c51b5ffb449b0554a94061"
server_key_iv: "d0a8746c68b319d8ca9320f3957f810e"
server_key_version: "encryptionkey@0"
I20260808 12:58:07.314502 7504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 12:58:07.317998 7531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:07.319396 7504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 12:58:07.319593 7504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "03a4137e9f9648e696d2f24066f5fe95"
format_stamp: "Formatted at 2026-08-08 12:58:07 on dist-test-slave-tk9z"
server_key: "49d430ac54c51b5ffb449b0554a94061"
server_key_iv: "d0a8746c68b319d8ca9320f3957f810e"
server_key_version: "encryptionkey@0"
I20260808 12:58:07.319861 7504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:58:07.326016 7504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:58:07.326817 7504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:58:07.327101 7504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:58:07.328008 7504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:58:07.329420 7504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:58:07.329547 7504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:07.329679 7504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:58:07.329767 7504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:07.363775 7504 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:41085
I20260808 12:58:07.363833 7704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:41085 every 8 connection(s)
I20260808 12:58:07.365165 7504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:58:07.371201 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 7504
I20260808 12:58:07.371413 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:58:07.372287 25244 external_mini_cluster.cc:1428] Setting key 63fe1a867eef3175d16eb12f7e836a4b
I20260808 12:58:07.373992 7708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:07.367811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:46073 (local address 127.24.167.62:35431)
0808 12:58:07.368254 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:07.368259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:07.369519 (+ 1260us) server_negotiation.cc:407] Connection header received
0808 12:58:07.371361 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:07.371367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:07.371449 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:07.371584 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:07.373161 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:07.373169 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:07.373177 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:58:07.373201 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:07.373266 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:58:07.373282 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:07.373387 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:07.373633 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 12:58:07.373695 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":230,"threads_started":1}
I20260808 12:58:07.375183 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:07.368430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:35431 (local address 127.24.167.1:46073)
0808 12:58:07.369351 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:07.369392 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:07.371071 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:07.371869 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:07.371881 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:07.372028 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:07.372790 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 12:58:07.372794 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:07.372842 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 12:58:07.372974 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:58:07.373387 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:58:07.373399 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:58:07.373406 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 12:58:07.373724 (+ 318us) client_negotiation.cc:240] Negotiation successful
0808 12:58:07.373978 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":274,"threads_started":1}
I20260808 12:58:07.375499 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35431
--builtin_ntp_servers=127.24.167.20:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 12:58:07.376868 7705 heartbeater.cc:347] Connected to a master server at 127.24.167.62:35431
I20260808 12:58:07.377184 7705 heartbeater.cc:464] Registering TS with master...
I20260808 12:58:07.377753 7705 heartbeater.cc:511] Master 127.24.167.62:35431 requested a full tablet report, sending...
I20260808 12:58:07.380040 7419 ts_manager.cc:194] Registered new tserver with Master: 03a4137e9f9648e696d2f24066f5fe95 (127.24.167.1:41085)
W20260808 12:58:07.720731 7710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:58:07.721128 7710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:58:07.721309 7710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:58:07.728725 7710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:58:07.728833 7710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:58:07.728950 7710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:58:07.738979 7710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:58:07.740329 7710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:58:07.742100 7710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:58:07.755842 7720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:07.755800 7719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:07.755856 7722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:58:07.756958 7710 server_base.cc:1034] running on GCE node
I20260808 12:58:07.757594 7710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:58:07.758621 7710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:58:07.759881 7710 hybrid_clock.cc:648] HybridClock initialized: now 1786193887759806 us; error 64 us; skew 500 ppm
I20260808 12:58:07.760185 7710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:58:07.762596 7710 webserver.cc:460] Webserver started at http://127.24.167.2:34297/ using document root <none> and password file <none>
I20260808 12:58:07.763381 7710 fs_manager.cc:359] Metadata directory not provided
I20260808 12:58:07.763478 7710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:58:07.763751 7710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:58:07.766767 7710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "6fa714a928aa48179ee84bef73cde140"
format_stamp: "Formatted at 2026-08-08 12:58:07 on dist-test-slave-tk9z"
server_key: "84de019ffeaedc0d7028b68e9b94fb78"
server_key_iv: "74023d8343d61a0e707145b4d0839b7f"
server_key_version: "encryptionkey@0"
I20260808 12:58:07.767549 7710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "6fa714a928aa48179ee84bef73cde140"
format_stamp: "Formatted at 2026-08-08 12:58:07 on dist-test-slave-tk9z"
server_key: "84de019ffeaedc0d7028b68e9b94fb78"
server_key_iv: "74023d8343d61a0e707145b4d0839b7f"
server_key_version: "encryptionkey@0"
I20260808 12:58:07.772761 7710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 12:58:07.776433 7733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:07.777806 7710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 12:58:07.778005 7710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "6fa714a928aa48179ee84bef73cde140"
format_stamp: "Formatted at 2026-08-08 12:58:07 on dist-test-slave-tk9z"
server_key: "84de019ffeaedc0d7028b68e9b94fb78"
server_key_iv: "74023d8343d61a0e707145b4d0839b7f"
server_key_version: "encryptionkey@0"
I20260808 12:58:07.778218 7710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:58:07.784200 7710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:58:07.784950 7710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:58:07.785221 7710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:58:07.786161 7710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:58:07.787712 7710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:58:07.787832 7710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:07.787973 7710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:58:07.788065 7710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:07.822332 7710 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:40253
I20260808 12:58:07.822474 7903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:40253 every 8 connection(s)
I20260808 12:58:07.823822 7710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:58:07.824209 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 7710
I20260808 12:58:07.824391 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:58:07.825101 25244 external_mini_cluster.cc:1428] Setting key aef42bb5d484f6275a029ca4b1bed152
I20260808 12:58:07.828225 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35431
--builtin_ntp_servers=127.24.167.20:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 12:58:07.832911 7708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:07.828044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:35065 (local address 127.24.167.62:35431)
0808 12:58:07.828223 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:07.828229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:07.828754 (+ 525us) server_negotiation.cc:407] Connection header received
0808 12:58:07.830599 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:07.830605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:07.830689 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:07.830815 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:07.832163 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:07.832168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:07.832173 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:58:07.832193 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:07.832237 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:58:07.832249 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:07.832374 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:07.832573 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 12:58:07.832643 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 12:58:07.833840 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:07.827494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:35431 (local address 127.24.167.2:35065)
0808 12:58:07.828571 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:07.828618 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:07.830231 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:07.831062 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:07.831071 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:07.831161 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:07.831860 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 12:58:07.831863 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:07.831900 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 12:58:07.831990 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:58:07.832346 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:58:07.832357 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:58:07.832364 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 12:58:07.832657 (+ 293us) client_negotiation.cc:240] Negotiation successful
0808 12:58:07.832906 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":657,"threads_started":1}
I20260808 12:58:07.835322 7904 heartbeater.cc:347] Connected to a master server at 127.24.167.62:35431
I20260808 12:58:07.835623 7904 heartbeater.cc:464] Registering TS with master...
I20260808 12:58:07.836215 7904 heartbeater.cc:511] Master 127.24.167.62:35431 requested a full tablet report, sending...
I20260808 12:58:07.837711 7419 ts_manager.cc:194] Registered new tserver with Master: 6fa714a928aa48179ee84bef73cde140 (127.24.167.2:40253)
W20260808 12:58:08.172597 7908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:58:08.172916 7908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:58:08.173077 7908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:58:08.180685 7908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:58:08.180815 7908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:58:08.180998 7908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:58:08.190872 7908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:58:08.192303 7908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:58:08.194175 7908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:58:08.207329 7916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:08.207563 7915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:08.207293 7919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:58:08.208212 7908 server_base.cc:1034] running on GCE node
I20260808 12:58:08.208899 7908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:58:08.209923 7908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:58:08.211198 7908 hybrid_clock.cc:648] HybridClock initialized: now 1786193888211137 us; error 48 us; skew 500 ppm
I20260808 12:58:08.211567 7908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 12:58:08.214093 7908 webserver.cc:460] Webserver started at http://127.24.167.3:38349/ using document root <none> and password file <none>
I20260808 12:58:08.214912 7908 fs_manager.cc:359] Metadata directory not provided
I20260808 12:58:08.215061 7908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:58:08.215404 7908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:58:08.218708 7908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "fd0f2e5b18b040af934f5b1551fd9f41"
format_stamp: "Formatted at 2026-08-08 12:58:08 on dist-test-slave-tk9z"
server_key: "65532fa650bcfb27f0555bb5275b149a"
server_key_iv: "fe47713dc4728775dcce3b6b18acd462"
server_key_version: "encryptionkey@0"
I20260808 12:58:08.219573 7908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "fd0f2e5b18b040af934f5b1551fd9f41"
format_stamp: "Formatted at 2026-08-08 12:58:08 on dist-test-slave-tk9z"
server_key: "65532fa650bcfb27f0555bb5275b149a"
server_key_iv: "fe47713dc4728775dcce3b6b18acd462"
server_key_version: "encryptionkey@0"
I20260808 12:58:08.225557 7908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 12:58:08.229444 7931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:08.231437 7908 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 12:58:08.231596 7908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "fd0f2e5b18b040af934f5b1551fd9f41"
format_stamp: "Formatted at 2026-08-08 12:58:08 on dist-test-slave-tk9z"
server_key: "65532fa650bcfb27f0555bb5275b149a"
server_key_iv: "fe47713dc4728775dcce3b6b18acd462"
server_key_version: "encryptionkey@0"
I20260808 12:58:08.231806 7908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:58:08.238509 7908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:58:08.239387 7908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:58:08.239647 7908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:58:08.240621 7908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:58:08.242120 7908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:58:08.242224 7908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:08.242308 7908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:58:08.242460 7908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:08.278672 7908 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:33859
I20260808 12:58:08.278723 8112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:33859 every 8 connection(s)
I20260808 12:58:08.280037 7908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:58:08.287402 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 7908
I20260808 12:58:08.287587 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:58:08.288074 25244 external_mini_cluster.cc:1428] Setting key 4f79058c7a96d10dda7f719f0d713eb0
I20260808 12:58:08.288693 7708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:08.282951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:42445 (local address 127.24.167.62:35431)
0808 12:58:08.283132 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:08.283136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:08.284502 (+ 1366us) server_negotiation.cc:407] Connection header received
0808 12:58:08.286213 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:08.286219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:08.286296 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:08.286462 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:08.287848 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:08.287854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:08.287858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:58:08.287884 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:08.287919 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:58:08.287933 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:08.288054 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:08.288314 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 12:58:08.288389 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 12:58:08.289738 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:08.283452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:35431 (local address 127.24.167.3:42445)
0808 12:58:08.284341 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:08.284375 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:08.285944 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:08.286674 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:08.286682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:08.286768 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:08.287484 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 12:58:08.287486 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:08.287524 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 12:58:08.287658 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:58:08.288041 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:58:08.288051 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:58:08.288058 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 12:58:08.288341 (+ 283us) client_negotiation.cc:240] Negotiation successful
0808 12:58:08.288531 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":267,"threads_started":1}
I20260808 12:58:08.291322 8113 heartbeater.cc:347] Connected to a master server at 127.24.167.62:35431
I20260808 12:58:08.291639 8113 heartbeater.cc:464] Registering TS with master...
I20260808 12:58:08.292238 8113 heartbeater.cc:511] Master 127.24.167.62:35431 requested a full tablet report, sending...
I20260808 12:58:08.293607 7419 ts_manager.cc:194] Registered new tserver with Master: fd0f2e5b18b040af934f5b1551fd9f41 (127.24.167.3:33859)
I20260808 12:58:08.303884 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 12:58:08.309948 7708 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 12:58:08.308496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.24.167.62:35431)
0808 12:58:08.309085 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:08.309090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:08.309196 (+ 106us) server_negotiation.cc:407] Connection header received
0808 12:58:08.309331 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:08.309334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:08.309387 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:08.309488 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:08.309753 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34416: BlockingRecv error: recv got EOF from 127.0.0.1:34416 (error 108)
Metrics: {"server-negotiator.queue_time_us":481}
I20260808 12:58:08.310617 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 7504
I20260808 12:58:08.314164 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:08.311664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.24.167.1:41085)
0808 12:58:08.312182 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:08.312190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:08.312215 (+ 25us) server_negotiation.cc:407] Connection header received
0808 12:58:08.312272 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:08.312277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:08.312484 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:08.312605 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:08.313517 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:08.313523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:08.313534 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:58:08.313545 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:08.313586 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:58:08.313600 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:08.313705 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:08.313827 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 12:58:08.313916 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":280,"threads_started":1}
I20260808 12:58:08.315483 7697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:58:08.398141 7705 heartbeater.cc:503] Master 127.24.167.62:35431 was elected leader, sending a full tablet report...
I20260808 12:58:08.399595 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 7504
W20260808 12:58:08.422677 7388 connection.cc:535] server connection from 127.24.167.1:46073 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 12:58:08.423089 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 7710
I20260808 12:58:08.427132 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:08.424236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.24.167.2:40253)
0808 12:58:08.424987 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:08.424994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:08.425022 (+ 28us) server_negotiation.cc:407] Connection header received
0808 12:58:08.425097 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:08.425103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:08.425329 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:08.425460 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:08.426475 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:08.426483 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:08.426495 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:58:08.426510 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:08.426563 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:58:08.426578 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:08.426690 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:08.426773 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 12:58:08.426862 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":436,"threads_started":1}
I20260808 12:58:08.428435 7895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:58:08.517215 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 7710
I20260808 12:58:08.541795 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 7908
I20260808 12:58:08.546556 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:08.542858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.24.167.3:33859)
0808 12:58:08.543544 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:08.543552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:08.543583 (+ 31us) server_negotiation.cc:407] Connection header received
0808 12:58:08.543658 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:08.543665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:08.543881 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:08.544061 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:08.545371 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:08.545382 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:08.545402 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 12:58:08.545425 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:08.545511 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 12:58:08.545534 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:08.545784 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:08.545990 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 12:58:08.546106 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":354,"threads_started":1}
I20260808 12:58:08.548496 8108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:58:08.631572 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 7908
I20260808 12:58:08.659451 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 7363
I20260808 12:58:08.660892 7464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:58:08.767529 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 7363
2026-08-08T12:58:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2528 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:58:08 dist-test-slave-tk9z krb5kdc[8147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:58:08 dist-test-slave-tk9z krb5kdc[8147](info): set up 2 sockets
Aug 08 12:58:08 dist-test-slave-tk9z krb5kdc[8147](info): commencing operation
krb5kdc: starting...
W20260808 12:58:10.119979 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.299s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:58:10 dist-test-slave-tk9z krb5kdc[8147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:58:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 12:58:10.387434 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:36755
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:36755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:58:10.730406 8171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:58:10.730754 8171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:58:10.730919 8171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:58:10.738206 8171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:58:10.738399 8171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:58:10.738508 8171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:58:10.738615 8171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:58:10.738691 8171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:58:10.749099 8171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:36755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:36755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:58:10.750701 8171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:58:10.752058 8171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:58:10.764017 8180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:10.764149 8184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:10.764232 8182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:58:10.765126 8171 server_base.cc:1034] running on GCE node
I20260808 12:58:10.765902 8171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:58:10.767359 8171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:58:10.768613 8171 hybrid_clock.cc:648] HybridClock initialized: now 1786193890768573 us; error 47 us; skew 500 ppm
I20260808 12:58:10.768966 8171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:58:10 dist-test-slave-tk9z krb5kdc[8147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193890, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:58:10.776660 8171 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:58:10.778264 8171 webserver.cc:460] Webserver started at http://127.24.167.62:46609/ using document root <none> and password file <none>
I20260808 12:58:10.779044 8171 fs_manager.cc:359] Metadata directory not provided
I20260808 12:58:10.779203 8171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:58:10.779541 8171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:58:10.782748 8171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "b02d9cb7253e49d1be65f2a4ee5bddc1"
format_stamp: "Formatted at 2026-08-08 12:58:10 on dist-test-slave-tk9z"
server_key: "3409c76db40d166de0dd129ee31f3197"
server_key_iv: "0b68f276066ffcc2d1fdc01693190d71"
server_key_version: "encryptionkey@0"
I20260808 12:58:10.783638 8171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "b02d9cb7253e49d1be65f2a4ee5bddc1"
format_stamp: "Formatted at 2026-08-08 12:58:10 on dist-test-slave-tk9z"
server_key: "3409c76db40d166de0dd129ee31f3197"
server_key_iv: "0b68f276066ffcc2d1fdc01693190d71"
server_key_version: "encryptionkey@0"
I20260808 12:58:10.788889 8171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 12:58:10.792546 8194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:10.793905 8171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 12:58:10.794094 8171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "b02d9cb7253e49d1be65f2a4ee5bddc1"
format_stamp: "Formatted at 2026-08-08 12:58:10 on dist-test-slave-tk9z"
server_key: "3409c76db40d166de0dd129ee31f3197"
server_key_iv: "0b68f276066ffcc2d1fdc01693190d71"
server_key_version: "encryptionkey@0"
I20260808 12:58:10.794301 8171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:58:10.812101 8171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:58:10.816706 8171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:58:10.817026 8171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:58:10.836134 8171 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:36755
I20260808 12:58:10.836179 8278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:36755 every 8 connection(s)
I20260808 12:58:10.837535 8171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:58:10.841308 8279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:58:10.844465 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 8171
I20260808 12:58:10.844705 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:58:10.845542 25244 external_mini_cluster.cc:1428] Setting key 1e23ed479e273c47caf738b4c9351bbd
I20260808 12:58:10.854880 8279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1: Bootstrap starting.
I20260808 12:58:10.858963 8279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1: Neither blocks nor log segments found. Creating new log.
I20260808 12:58:10.860460 8279 log.cc:824] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1: Log is configured to *not* fsync() on all Append() calls
Aug 08 12:58:10 dist-test-slave-tk9z krb5kdc[8147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:58:10.864181 8279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1: No bootstrap required, opened a new log
I20260808 12:58:10.870843 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:10.847808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.24.167.62:36755)
0808 12:58:10.848605 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:10.848617 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:10.848660 (+ 43us) server_negotiation.cc:407] Connection header received
0808 12:58:10.850481 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:10.850507 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:10.850936 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:10.851284 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:10.853210 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:10.854678 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:10.856576 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:10.857289 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:10.857423 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:10.862896 (+ 5473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:10.862905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:10.862922 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:58:10.862961 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:10.866298 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:10.867105 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:10.867112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:10.867118 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:10.867295 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:10.867737 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:10.867752 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:10.867754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:10.868398 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:10.868769 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:10.869220 (+ 451us) server_negotiation.cc:299] Negotiation successful
0808 12:58:10.869678 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":391,"threads_started":1}
I20260808 12:58:10.872615 8279 raft_consensus.cc:348] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d9cb7253e49d1be65f2a4ee5bddc1" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 36755 } }
I20260808 12:58:10.873028 8279 raft_consensus.cc:374] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:58:10.873157 8279 raft_consensus.cc:729] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b02d9cb7253e49d1be65f2a4ee5bddc1, State: Initialized, Role: FOLLOWER
I20260808 12:58:10.873762 8279 consensus_queue.cc:260] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d9cb7253e49d1be65f2a4ee5bddc1" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 36755 } }
I20260808 12:58:10.874063 8279 raft_consensus.cc:388] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:58:10.874218 8279 raft_consensus.cc:482] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:58:10.874403 8279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:58:10.876623 8279 raft_consensus.cc:504] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d9cb7253e49d1be65f2a4ee5bddc1" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 36755 } }
I20260808 12:58:10.877151 8279 leader_election.cc:302] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b02d9cb7253e49d1be65f2a4ee5bddc1; no voters:
I20260808 12:58:10.877737 8279 leader_election.cc:288] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:58:10.877883 8284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:58:10.878629 8284 raft_consensus.cc:686] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [term 1 LEADER]: Becoming Leader. State: Replica: b02d9cb7253e49d1be65f2a4ee5bddc1, State: Running, Role: LEADER
I20260808 12:58:10.879213 8284 consensus_queue.cc:237] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d9cb7253e49d1be65f2a4ee5bddc1" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 36755 } }
I20260808 12:58:10.879937 8279 sys_catalog.cc:564] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:58:10.882407 8288 sys_catalog.cc:455] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b02d9cb7253e49d1be65f2a4ee5bddc1. Latest consensus state: current_term: 1 leader_uuid: "b02d9cb7253e49d1be65f2a4ee5bddc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d9cb7253e49d1be65f2a4ee5bddc1" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 36755 } } }
I20260808 12:58:10.882609 8285 sys_catalog.cc:455] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b02d9cb7253e49d1be65f2a4ee5bddc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d9cb7253e49d1be65f2a4ee5bddc1" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 36755 } } }
I20260808 12:58:10.882867 8288 sys_catalog.cc:458] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [sys.catalog]: This master's current role is: LEADER
I20260808 12:58:10.882915 8285 sys_catalog.cc:458] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1 [sys.catalog]: This master's current role is: LEADER
I20260808 12:58:10.886062 8296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:58:10.891826 8296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:58:10.900931 8296 catalog_manager.cc:1347] Generated new cluster ID: def8ee21265b4a25803a95e8eeddfbe8
I20260808 12:58:10.901060 8296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:58:10.915670 8296 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:58:10.918253 8296 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:58:10.930974 8296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1: Generated new TSK 0
I20260808 12:58:10.932137 8296 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 12:58:11.002140 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:36755
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:58:11.352512 8320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:58:11.352818 8320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:58:11.352981 8320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:58:11.360702 8320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:58:11.360809 8320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:58:11.361008 8320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:58:11.371820 8320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:36755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:58:11.373188 8320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:58:11.374671 8320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:58:11.388074 8338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:11.388123 8328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:11.388031 8329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:58:11.388960 8320 server_base.cc:1034] running on GCE node
I20260808 12:58:11.389585 8320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:58:11.390612 8320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:58:11.391844 8320 hybrid_clock.cc:648] HybridClock initialized: now 1786193891391804 us; error 50 us; skew 500 ppm
I20260808 12:58:11.392210 8320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:58:11 dist-test-slave-tk9z krb5kdc[8147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193891, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:58:11.395488 8320 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 12:58:11.396950 8320 webserver.cc:460] Webserver started at http://127.24.167.1:36907/ using document root <none> and password file <none>
I20260808 12:58:11.397686 8320 fs_manager.cc:359] Metadata directory not provided
I20260808 12:58:11.397837 8320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:58:11.398236 8320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:58:11.401371 8320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "75c5a71873e04b238c87c1c6773d2750"
format_stamp: "Formatted at 2026-08-08 12:58:11 on dist-test-slave-tk9z"
server_key: "5247b0492fac225246fee6bc47ef7837"
server_key_iv: "029c13bc31798e17d224f8a7c980cb46"
server_key_version: "encryptionkey@0"
I20260808 12:58:11.402174 8320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "75c5a71873e04b238c87c1c6773d2750"
format_stamp: "Formatted at 2026-08-08 12:58:11 on dist-test-slave-tk9z"
server_key: "5247b0492fac225246fee6bc47ef7837"
server_key_iv: "029c13bc31798e17d224f8a7c980cb46"
server_key_version: "encryptionkey@0"
I20260808 12:58:11.407562 8320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 12:58:11.411381 8349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:11.412833 8320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 12:58:11.413029 8320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "75c5a71873e04b238c87c1c6773d2750"
format_stamp: "Formatted at 2026-08-08 12:58:11 on dist-test-slave-tk9z"
server_key: "5247b0492fac225246fee6bc47ef7837"
server_key_iv: "029c13bc31798e17d224f8a7c980cb46"
server_key_version: "encryptionkey@0"
I20260808 12:58:11.413255 8320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:58:11.447396 8320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:58:11.451759 8320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:58:11.452082 8320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:58:11.453125 8320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:58:11.454577 8320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:58:11.454680 8320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:11.454849 8320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:58:11.454922 8320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:11.488581 8320 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:37595
I20260808 12:58:11.488628 8529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:37595 every 8 connection(s)
I20260808 12:58:11.489931 8320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:58:11.491745 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 8320
I20260808 12:58:11.491969 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:58:11.492762 25244 external_mini_cluster.cc:1428] Setting key 786d9a63058608786cd4cc966dc5521d
Aug 08 12:58:11 dist-test-slave-tk9z krb5kdc[8147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193891, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:58:11.511341 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:11.492639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:40177 (local address 127.24.167.62:36755)
0808 12:58:11.493230 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:11.493236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:11.494261 (+ 1025us) server_negotiation.cc:407] Connection header received
0808 12:58:11.496145 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:11.496154 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:11.496260 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:11.496387 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:11.497990 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:11.499171 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:11.500778 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:11.501481 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:11.501628 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:11.505307 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:11.505312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:11.505315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:58:11.505347 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:11.508159 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:11.508914 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:11.508919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:11.508925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:11.509053 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:11.509497 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:11.509502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:11.509505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:11.509776 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:11.509939 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:11.510732 (+ 793us) server_negotiation.cc:299] Negotiation successful
0808 12:58:11.510969 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":309,"threads_started":1}
I20260808 12:58:11.512358 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:11.493251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:36755 (local address 127.24.167.1:40177)
0808 12:58:11.494088 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:11.494141 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:11.495876 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:11.496596 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:11.496605 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:11.497197 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:11.497817 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:11.497837 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:11.499349 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:11.499353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:11.500620 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:11.500632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:11.501791 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:11.501797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:11.501997 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:11.502725 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:58:11.502761 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:11.505068 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:58:11.508324 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:58:11.508333 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:58:11.508350 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:58:11.508780 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:58:11.509168 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:58:11.509173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:58:11.509176 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:58:11.509372 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:58:11.509910 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:58:11.509919 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:58:11.510289 (+ 370us) client_negotiation.cc:769] Sending connection context
0808 12:58:11.510642 (+ 353us) client_negotiation.cc:240] Negotiation successful
0808 12:58:11.511202 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":341,"threads_started":1}
I20260808 12:58:11.514010 8530 heartbeater.cc:347] Connected to a master server at 127.24.167.62:36755
I20260808 12:58:11.514326 8530 heartbeater.cc:464] Registering TS with master...
I20260808 12:58:11.515136 8530 heartbeater.cc:511] Master 127.24.167.62:36755 requested a full tablet report, sending...
I20260808 12:58:11.517750 8223 ts_manager.cc:194] Registered new tserver with Master: 75c5a71873e04b238c87c1c6773d2750 (127.24.167.1:37595)
I20260808 12:58:11.520165 8223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:40177
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 12:58:11.555646 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:36755
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:58:11.897017 8539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:58:11.897388 8539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:58:11.897579 8539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:58:11.905167 8539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:58:11.905290 8539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:58:11.905470 8539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:58:11.915632 8539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:36755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:58:11.917076 8539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:58:11.918525 8539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:58:11.932300 8551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:11.932363 8553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:11.932413 8550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:58:11.933013 8539 server_base.cc:1034] running on GCE node
I20260808 12:58:11.933799 8539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:58:11.934917 8539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:58:11.936244 8539 hybrid_clock.cc:648] HybridClock initialized: now 1786193891936135 us; error 43 us; skew 500 ppm
I20260808 12:58:11.936641 8539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:58:11 dist-test-slave-tk9z krb5kdc[8147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193891, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:58:11.940861 8539 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 12:58:11.944367 8539 webserver.cc:460] Webserver started at http://127.24.167.2:44739/ using document root <none> and password file <none>
I20260808 12:58:11.945176 8539 fs_manager.cc:359] Metadata directory not provided
I20260808 12:58:11.945350 8539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:58:11.945701 8539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:58:11.949147 8539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "a5512aa690674c0f8e1086289a63b5fd"
format_stamp: "Formatted at 2026-08-08 12:58:11 on dist-test-slave-tk9z"
server_key: "015832798156f0b0c741a31b1b34f395"
server_key_iv: "9dd4934e788526f1977197fc11927a8b"
server_key_version: "encryptionkey@0"
I20260808 12:58:11.950047 8539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "a5512aa690674c0f8e1086289a63b5fd"
format_stamp: "Formatted at 2026-08-08 12:58:11 on dist-test-slave-tk9z"
server_key: "015832798156f0b0c741a31b1b34f395"
server_key_iv: "9dd4934e788526f1977197fc11927a8b"
server_key_version: "encryptionkey@0"
I20260808 12:58:11.955829 8539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 12:58:11.959636 8561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:11.961192 8539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 12:58:11.961375 8539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "a5512aa690674c0f8e1086289a63b5fd"
format_stamp: "Formatted at 2026-08-08 12:58:11 on dist-test-slave-tk9z"
server_key: "015832798156f0b0c741a31b1b34f395"
server_key_iv: "9dd4934e788526f1977197fc11927a8b"
server_key_version: "encryptionkey@0"
I20260808 12:58:11.961597 8539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:58:11.997859 8539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:58:12.002933 8539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:58:12.003226 8539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:58:12.004261 8539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:58:12.005731 8539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:58:12.005816 8539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:12.005995 8539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:58:12.006049 8539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:12.041055 8539 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:37333
I20260808 12:58:12.041173 8742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:37333 every 8 connection(s)
I20260808 12:58:12.042439 8539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:58:12.045454 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 8539
I20260808 12:58:12.045642 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:58:12.046231 25244 external_mini_cluster.cc:1428] Setting key 2b721853ab7cda9aed6b8931311ed9bf
Aug 08 12:58:12 dist-test-slave-tk9z krb5kdc[8147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193891, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:58:12.066497 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:12.045147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:37171 (local address 127.24.167.62:36755)
0808 12:58:12.045693 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:12.045702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:12.046838 (+ 1136us) server_negotiation.cc:407] Connection header received
0808 12:58:12.049112 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:12.049116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:12.049182 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:12.049278 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:12.051522 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.052287 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.054763 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.055191 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.055288 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:12.059653 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:12.059658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:12.059662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:58:12.059695 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:12.062982 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:12.063772 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:12.063776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:12.063778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:12.063918 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:12.064391 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:12.064394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:12.064397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:12.064685 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:12.064866 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:12.065864 (+ 998us) server_negotiation.cc:299] Negotiation successful
0808 12:58:12.066071 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":262,"threads_started":1}
I20260808 12:58:12.067982 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:12.045848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:36755 (local address 127.24.167.2:37171)
0808 12:58:12.046674 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:12.046718 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:12.048781 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:12.049565 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:12.049579 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:12.050470 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:12.051330 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:12.051360 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.052449 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.052458 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:12.054617 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:12.054636 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.055420 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.055429 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:12.055768 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:12.056533 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:58:12.056569 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:12.059408 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:58:12.063139 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:58:12.063148 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:58:12.063167 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:58:12.063627 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:58:12.064040 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:58:12.064045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:58:12.064049 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:58:12.064256 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:58:12.064889 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:58:12.064900 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:58:12.065406 (+ 506us) client_negotiation.cc:769] Sending connection context
0808 12:58:12.065748 (+ 342us) client_negotiation.cc:240] Negotiation successful
0808 12:58:12.066407 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":332,"threads_started":1}
I20260808 12:58:12.069542 8743 heartbeater.cc:347] Connected to a master server at 127.24.167.62:36755
I20260808 12:58:12.069891 8743 heartbeater.cc:464] Registering TS with master...
I20260808 12:58:12.070919 8743 heartbeater.cc:511] Master 127.24.167.62:36755 requested a full tablet report, sending...
I20260808 12:58:12.072811 8223 ts_manager.cc:194] Registered new tserver with Master: a5512aa690674c0f8e1086289a63b5fd (127.24.167.2:37333)
I20260808 12:58:12.073772 8223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:37171
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 12:58:12.112201 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:36755
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:58:12.459721 8754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:58:12.460110 8754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:58:12.460213 8754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:58:12.467501 8754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:58:12.467610 8754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:58:12.467729 8754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:58:12.477481 8754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:36755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:58:12.478848 8754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:58:12.480139 8754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:58:12.493585 8761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:12.493827 8764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:58:12.494107 8760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:58:12.495428 8754 server_base.cc:1034] running on GCE node
I20260808 12:58:12.496068 8754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:58:12.497013 8754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:58:12.498250 8754 hybrid_clock.cc:648] HybridClock initialized: now 1786193892498212 us; error 33 us; skew 500 ppm
I20260808 12:58:12.498683 8754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:58:12 dist-test-slave-tk9z krb5kdc[8147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193892, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:58:12.501858 8754 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 12:58:12.503306 8754 webserver.cc:460] Webserver started at http://127.24.167.3:33107/ using document root <none> and password file <none>
I20260808 12:58:12.504029 8754 fs_manager.cc:359] Metadata directory not provided
I20260808 12:58:12.504179 8754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:58:12.504501 8754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 12:58:12.507498 8754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2"
format_stamp: "Formatted at 2026-08-08 12:58:12 on dist-test-slave-tk9z"
server_key: "a2a22374af2bead32a477f4a876289cb"
server_key_iv: "052f34a23463d2e019b8a71f4a85159f"
server_key_version: "encryptionkey@0"
I20260808 12:58:12.508272 8754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2"
format_stamp: "Formatted at 2026-08-08 12:58:12 on dist-test-slave-tk9z"
server_key: "a2a22374af2bead32a477f4a876289cb"
server_key_iv: "052f34a23463d2e019b8a71f4a85159f"
server_key_version: "encryptionkey@0"
I20260808 12:58:12.513427 8754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 12:58:12.516906 8778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:12.518216 8754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 12:58:12.518455 8754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2"
format_stamp: "Formatted at 2026-08-08 12:58:12 on dist-test-slave-tk9z"
server_key: "a2a22374af2bead32a477f4a876289cb"
server_key_iv: "052f34a23463d2e019b8a71f4a85159f"
server_key_version: "encryptionkey@0"
I20260808 12:58:12.518668 8754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:58:12.523834 8530 heartbeater.cc:503] Master 127.24.167.62:36755 was elected leader, sending a full tablet report...
I20260808 12:58:12.547473 8754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:58:12.552175 8754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:58:12.552538 8754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:58:12.553578 8754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:58:12.555251 8754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:58:12.555351 8754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:12.555501 8754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:58:12.555567 8754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:58:12.590770 8754 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:39931
I20260808 12:58:12.590816 8971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:39931 every 8 connection(s)
I20260808 12:58:12.592088 8754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:58:12.592551 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 8754
I20260808 12:58:12.592747 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:58:12.593396 25244 external_mini_cluster.cc:1428] Setting key 8888095e8501c0f9006d5560ad48a3e1
Aug 08 12:58:12 dist-test-slave-tk9z krb5kdc[8147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193892, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:58:12.612931 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:12.595018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:36113 (local address 127.24.167.62:36755)
0808 12:58:12.595665 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:12.595669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:12.596667 (+ 998us) server_negotiation.cc:407] Connection header received
0808 12:58:12.598649 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:12.598653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:12.598790 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:12.598897 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:12.600389 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.601188 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.602837 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.603256 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.603353 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:12.606780 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:12.606785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:12.606788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:58:12.606820 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:12.610045 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:12.610846 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:12.610850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:12.610853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:12.610991 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:12.611384 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:12.611388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:12.611390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:12.611665 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:12.611787 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:12.612464 (+ 677us) server_negotiation.cc:299] Negotiation successful
0808 12:58:12.612637 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":228,"threads_started":1}
I20260808 12:58:12.613929 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:12.595598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:36755 (local address 127.24.167.3:36113)
0808 12:58:12.596507 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:12.596554 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:12.598329 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:12.599120 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:12.599129 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:12.599659 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:58:12.600238 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:12.600255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.601329 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.601333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:12.602703 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:12.602716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.603457 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.603462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:12.603665 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:12.604305 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:58:12.604327 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:58:12.606583 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:58:12.610181 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:58:12.610189 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:58:12.610202 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:58:12.610733 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:58:12.611087 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:58:12.611091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:58:12.611093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:58:12.611282 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:58:12.611787 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:58:12.611793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:58:12.612136 (+ 343us) client_negotiation.cc:769] Sending connection context
0808 12:58:12.612370 (+ 234us) client_negotiation.cc:240] Negotiation successful
0808 12:58:12.612773 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":386,"threads_started":1}
I20260808 12:58:12.615283 8972 heartbeater.cc:347] Connected to a master server at 127.24.167.62:36755
I20260808 12:58:12.615605 8972 heartbeater.cc:464] Registering TS with master...
I20260808 12:58:12.616356 8972 heartbeater.cc:511] Master 127.24.167.62:36755 requested a full tablet report, sending...
I20260808 12:58:12.617960 8223 ts_manager.cc:194] Registered new tserver with Master: 4c1db5c4ab0241f8bb63d5cce877b6d2 (127.24.167.3:39931)
I20260808 12:58:12.618834 8223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:36113
I20260808 12:58:12.621822 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 12:58:12.639123 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:12.626770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.24.167.62:36755)
0808 12:58:12.626942 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:12.626955 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:12.627529 (+ 574us) server_negotiation.cc:407] Connection header received
0808 12:58:12.627678 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:12.627682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:12.627746 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:12.627834 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:12.629203 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.629936 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.631242 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.631662 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.631757 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:12.634004 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:12.634009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:12.634012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:58:12.634046 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:12.636757 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:12.637371 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:12.637374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:12.637377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:12.637507 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:12.637801 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:12.637804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:12.637806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:12.638153 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:12.638334 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:12.638716 (+ 382us) server_negotiation.cc:299] Negotiation successful
0808 12:58:12.638885 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 12:58:12.643332 8223 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:58:12.650293 8223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:58:12.686115 9000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:12.671320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45388 (local address 127.24.167.2:37333)
0808 12:58:12.678401 (+ 7081us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:12.678412 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:12.678454 (+ 42us) server_negotiation.cc:407] Connection header received
0808 12:58:12.678537 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:12.678542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:12.678696 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:12.678900 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:58:12.679608 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.681097 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.683556 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.684857 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.685035 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:12.685269 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:12.685630 (+ 361us) server_negotiation.cc:299] Negotiation successful
0808 12:58:12.685795 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6902,"thread_start_us":6788,"threads_started":1}
I20260808 12:58:12.686302 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:12.676652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:37333 (local address 127.0.0.1:45388)
0808 12:58:12.677697 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:12.677720 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:12.677870 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:12.678889 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:12.678894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:12.678920 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:58:12.679379 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:12.679395 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.681223 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.681229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:12.683367 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:12.683385 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.685157 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.685164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:12.685376 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:12.685415 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 12:58:12.685571 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 12:58:12.685768 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":25,"thread_start_us":747,"threads_started":1}
I20260808 12:58:12.687054 8993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:12.663404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:39931 (local address 127.0.0.1:42460)
0808 12:58:12.676805 (+ 13401us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:12.676833 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:12.677022 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:12.678334 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:12.678341 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:12.678392 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:58:12.678824 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:12.678838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.681134 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.681143 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:12.683712 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:12.683733 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.685998 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.686008 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:12.686233 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:12.686272 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 12:58:12.686348 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 12:58:12.686518 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13219,"mutex_wait_us":26,"thread_start_us":13004,"threads_started":1}
I20260808 12:58:12.687078 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:12.662779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37595 (local address 127.0.0.1:55764)
0808 12:58:12.670206 (+ 7427us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:12.670300 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:12.670582 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:12.679300 (+ 8718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:12.679306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:12.679336 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:58:12.679799 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:12.679813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.681903 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.681909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:12.684133 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:12.684146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.686127 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.686135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:12.686301 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:12.686343 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 12:58:12.686414 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 12:58:12.686518 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7042,"mutex_wait_us":4,"thread_start_us":6844,"threads_started":1}
I20260808 12:58:12.687994 8992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:12.663005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.24.167.1:37595)
0808 12:58:12.678469 (+ 15464us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:12.678479 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:12.678511 (+ 32us) server_negotiation.cc:407] Connection header received
0808 12:58:12.678621 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:12.678627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:12.678854 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:12.679995 (+ 1141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:58:12.680238 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.681749 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.684316 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.685864 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.687146 (+ 1282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:12.687412 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:12.687493 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 12:58:12.687678 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15239,"thread_start_us":16015,"threads_started":1}
I20260808 12:58:12.689148 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:12.663745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42460 (local address 127.24.167.3:39931)
0808 12:58:12.675983 (+ 12238us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:12.675998 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:12.677190 (+ 1192us) server_negotiation.cc:407] Connection header received
0808 12:58:12.677349 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:12.677362 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:12.677724 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:12.677988 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:58:12.679106 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.680994 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.683906 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:12.685470 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:12.685611 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:12.685829 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:12.688610 (+ 2781us) server_negotiation.cc:299] Negotiation successful
0808 12:58:12.688801 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11947,"thread_start_us":12257,"threads_started":1}
I20260808 12:58:12.691112 8640 tablet_service.cc:1467] Processing CreateTablet for tablet ca8f318a70a441dbb41baf282e013b5d (DEFAULT_TABLE table=test-table [id=b5512813a685435f8f18a9026898ff2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:58:12.691205 8419 tablet_service.cc:1467] Processing CreateTablet for tablet ca8f318a70a441dbb41baf282e013b5d (DEFAULT_TABLE table=test-table [id=b5512813a685435f8f18a9026898ff2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:58:12.692462 8640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8f318a70a441dbb41baf282e013b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:58:12.692633 8419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8f318a70a441dbb41baf282e013b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:58:12.692880 8859 tablet_service.cc:1467] Processing CreateTablet for tablet ca8f318a70a441dbb41baf282e013b5d (DEFAULT_TABLE table=test-table [id=b5512813a685435f8f18a9026898ff2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:58:12.694324 8859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8f318a70a441dbb41baf282e013b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:58:12.704514 9004 tablet_bootstrap.cc:492] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd: Bootstrap starting.
I20260808 12:58:12.708335 9004 tablet_bootstrap.cc:654] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd: Neither blocks nor log segments found. Creating new log.
I20260808 12:58:12.709424 9004 log.cc:824] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd: Log is configured to *not* fsync() on all Append() calls
I20260808 12:58:12.714741 9005 tablet_bootstrap.cc:492] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750: Bootstrap starting.
I20260808 12:58:12.718797 9006 tablet_bootstrap.cc:492] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2: Bootstrap starting.
I20260808 12:58:12.719506 9005 tablet_bootstrap.cc:654] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750: Neither blocks nor log segments found. Creating new log.
I20260808 12:58:12.721091 9005 log.cc:824] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750: Log is configured to *not* fsync() on all Append() calls
I20260808 12:58:12.722769 9006 tablet_bootstrap.cc:654] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2: Neither blocks nor log segments found. Creating new log.
I20260808 12:58:12.723193 9004 tablet_bootstrap.cc:492] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd: No bootstrap required, opened a new log
I20260808 12:58:12.723513 9004 ts_tablet_manager.cc:1397] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.012s
I20260808 12:58:12.724094 9006 log.cc:824] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2: Log is configured to *not* fsync() on all Append() calls
I20260808 12:58:12.724418 9005 tablet_bootstrap.cc:492] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750: No bootstrap required, opened a new log
I20260808 12:58:12.724712 9005 ts_tablet_manager.cc:1397] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260808 12:58:12.727981 9006 tablet_bootstrap.cc:492] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2: No bootstrap required, opened a new log
I20260808 12:58:12.728253 9006 ts_tablet_manager.cc:1397] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260808 12:58:12.732606 9005 raft_consensus.cc:348] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c5a71873e04b238c87c1c6773d2750" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37595 } } peers { permanent_uuid: "a5512aa690674c0f8e1086289a63b5fd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 37333 } } peers { permanent_uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39931 } }
I20260808 12:58:12.733000 9005 raft_consensus.cc:374] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:58:12.733095 9005 raft_consensus.cc:729] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c5a71873e04b238c87c1c6773d2750, State: Initialized, Role: FOLLOWER
I20260808 12:58:12.733736 9005 consensus_queue.cc:260] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c5a71873e04b238c87c1c6773d2750" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37595 } } peers { permanent_uuid: "a5512aa690674c0f8e1086289a63b5fd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 37333 } } peers { permanent_uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39931 } }
I20260808 12:58:12.735378 9005 ts_tablet_manager.cc:1428] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260808 12:58:12.734517 9004 raft_consensus.cc:348] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c5a71873e04b238c87c1c6773d2750" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37595 } } peers { permanent_uuid: "a5512aa690674c0f8e1086289a63b5fd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 37333 } } peers { permanent_uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39931 } }
I20260808 12:58:12.735822 9004 raft_consensus.cc:374] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:58:12.735961 9004 raft_consensus.cc:729] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5512aa690674c0f8e1086289a63b5fd, State: Initialized, Role: FOLLOWER
I20260808 12:58:12.736596 9004 consensus_queue.cc:260] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c5a71873e04b238c87c1c6773d2750" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37595 } } peers { permanent_uuid: "a5512aa690674c0f8e1086289a63b5fd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 37333 } } peers { permanent_uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39931 } }
I20260808 12:58:12.738185 8743 heartbeater.cc:503] Master 127.24.167.62:36755 was elected leader, sending a full tablet report...
I20260808 12:58:12.738399 9004 ts_tablet_manager.cc:1428] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260808 12:58:12.738545 9006 raft_consensus.cc:348] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c5a71873e04b238c87c1c6773d2750" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37595 } } peers { permanent_uuid: "a5512aa690674c0f8e1086289a63b5fd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 37333 } } peers { permanent_uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39931 } }
I20260808 12:58:12.738997 9006 raft_consensus.cc:374] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:58:12.739104 9006 raft_consensus.cc:729] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1db5c4ab0241f8bb63d5cce877b6d2, State: Initialized, Role: FOLLOWER
I20260808 12:58:12.739684 9006 consensus_queue.cc:260] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c5a71873e04b238c87c1c6773d2750" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37595 } } peers { permanent_uuid: "a5512aa690674c0f8e1086289a63b5fd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 37333 } } peers { permanent_uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39931 } }
I20260808 12:58:12.741549 8972 heartbeater.cc:503] Master 127.24.167.62:36755 was elected leader, sending a full tablet report...
I20260808 12:58:12.742187 9006 ts_tablet_manager.cc:1428] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260808 12:58:12.742750 8531 tablet.cc:2369] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 12:58:12.794687 8745 tablet.cc:2369] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 12:58:12.844161 8975 tablet.cc:2369] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:58:12.996551 9015 raft_consensus.cc:482] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:58:12.996907 9015 raft_consensus.cc:504] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c5a71873e04b238c87c1c6773d2750" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37595 } } peers { permanent_uuid: "a5512aa690674c0f8e1086289a63b5fd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 37333 } } peers { permanent_uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39931 } }
I20260808 12:58:12.999145 9015 leader_election.cc:288] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75c5a71873e04b238c87c1c6773d2750 (127.24.167.1:37595), 4c1db5c4ab0241f8bb63d5cce877b6d2 (127.24.167.3:39931)
I20260808 12:58:13.006176 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:12.999888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:56591 (local address 127.24.167.3:39931)
0808 12:58:13.000159 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:13.000168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:13.000818 (+ 650us) server_negotiation.cc:407] Connection header received
0808 12:58:13.001031 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:13.001036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:13.001095 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:13.001194 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:58:13.001774 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.002562 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.004351 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.005145 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.005346 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:13.005452 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:13.005762 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 12:58:13.005895 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 12:58:13.006202 8992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:12.999525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:40407 (local address 127.24.167.1:37595)
0808 12:58:12.999807 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:12.999817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:13.000532 (+ 715us) server_negotiation.cc:407] Connection header received
0808 12:58:13.000785 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:13.000794 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:13.000857 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:13.000959 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:58:13.001542 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.002296 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.004208 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.004998 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.005196 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:13.005355 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:13.005736 (+ 381us) server_negotiation.cc:299] Negotiation successful
0808 12:58:13.005907 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 12:58:13.006445 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:12.999536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37595 (local address 127.24.167.2:40407)
0808 12:58:13.000407 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:13.000422 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:13.000628 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:13.000982 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:13.000987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:13.001011 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:58:13.001379 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:13.001390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.002493 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.002504 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:13.004036 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:13.004049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.005228 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.005235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:13.005526 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:13.005559 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 12:58:13.005706 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 12:58:13.005927 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":63,"thread_start_us":845,"threads_started":1}
I20260808 12:58:13.006830 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:12.999578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:39931 (local address 127.24.167.2:56591)
0808 12:58:13.000706 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:13.000720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 12:58:13.000876 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:58:13.001213 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:58:13.001217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:58:13.001240 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:58:13.001614 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:13.001623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.002703 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.002709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:13.004175 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:58:13.004188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.005384 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.005390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:58:13.005600 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:13.005623 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 12:58:13.005730 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 12:58:13.005892 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"mutex_wait_us":3,"thread_start_us":468,"threads_started":1}
I20260808 12:58:13.007477 8455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8f318a70a441dbb41baf282e013b5d" candidate_uuid: "a5512aa690674c0f8e1086289a63b5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c5a71873e04b238c87c1c6773d2750" is_pre_election: true
I20260808 12:58:13.007874 8890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8f318a70a441dbb41baf282e013b5d" candidate_uuid: "a5512aa690674c0f8e1086289a63b5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2" is_pre_election: true
I20260808 12:58:13.008057 8455 raft_consensus.cc:2454] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5512aa690674c0f8e1086289a63b5fd in term 0.
I20260808 12:58:13.008395 8890 raft_consensus.cc:2454] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5512aa690674c0f8e1086289a63b5fd in term 0.
I20260808 12:58:13.008803 8567 leader_election.cc:302] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75c5a71873e04b238c87c1c6773d2750, a5512aa690674c0f8e1086289a63b5fd; no voters:
I20260808 12:58:13.009238 9015 raft_consensus.cc:2781] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:58:13.009398 9015 raft_consensus.cc:482] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:58:13.009508 9015 raft_consensus.cc:3037] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:58:13.011858 9015 raft_consensus.cc:504] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c5a71873e04b238c87c1c6773d2750" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37595 } } peers { permanent_uuid: "a5512aa690674c0f8e1086289a63b5fd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 37333 } } peers { permanent_uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39931 } }
I20260808 12:58:13.012475 9015 leader_election.cc:288] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [CANDIDATE]: Term 1 election: Requested vote from peers 75c5a71873e04b238c87c1c6773d2750 (127.24.167.1:37595), 4c1db5c4ab0241f8bb63d5cce877b6d2 (127.24.167.3:39931)
I20260808 12:58:13.012944 8455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8f318a70a441dbb41baf282e013b5d" candidate_uuid: "a5512aa690674c0f8e1086289a63b5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c5a71873e04b238c87c1c6773d2750"
I20260808 12:58:13.013056 8890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8f318a70a441dbb41baf282e013b5d" candidate_uuid: "a5512aa690674c0f8e1086289a63b5fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2"
I20260808 12:58:13.013154 8455 raft_consensus.cc:3037] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:58:13.013295 8890 raft_consensus.cc:3037] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:58:13.015506 8455 raft_consensus.cc:2454] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5512aa690674c0f8e1086289a63b5fd in term 1.
I20260808 12:58:13.015635 8890 raft_consensus.cc:2454] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5512aa690674c0f8e1086289a63b5fd in term 1.
I20260808 12:58:13.015934 8567 leader_election.cc:302] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75c5a71873e04b238c87c1c6773d2750, a5512aa690674c0f8e1086289a63b5fd; no voters:
I20260808 12:58:13.016259 9015 raft_consensus.cc:2781] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:58:13.016727 9015 raft_consensus.cc:686] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [term 1 LEADER]: Becoming Leader. State: Replica: a5512aa690674c0f8e1086289a63b5fd, State: Running, Role: LEADER
I20260808 12:58:13.017264 9015 consensus_queue.cc:237] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c5a71873e04b238c87c1c6773d2750" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37595 } } peers { permanent_uuid: "a5512aa690674c0f8e1086289a63b5fd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 37333 } } peers { permanent_uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39931 } }
I20260808 12:58:13.022258 8222 catalog_manager.cc:5495] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd reported cstate change: term changed from 0 to 1, leader changed from <none> to a5512aa690674c0f8e1086289a63b5fd (127.24.167.2). New cstate: current_term: 1 leader_uuid: "a5512aa690674c0f8e1086289a63b5fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c5a71873e04b238c87c1c6773d2750" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5512aa690674c0f8e1086289a63b5fd" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 37333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39931 } health_report { overall_health: UNKNOWN } } }
I20260808 12:58:13.056243 9000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:13.050176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45402 (local address 127.24.167.2:37333)
0808 12:58:13.050487 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:13.050493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:13.050516 (+ 23us) server_negotiation.cc:407] Connection header received
0808 12:58:13.050703 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:13.050708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:13.050827 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:13.050985 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:58:13.051302 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.052451 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.053812 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.054499 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.054616 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:13.055093 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:58:13.055727 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:58:13.055833 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:13.055977 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 12:58:13.056043 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 12:58:13.064260 8890 raft_consensus.cc:1261] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2 [term 1 FOLLOWER]: Refusing update from remote peer a5512aa690674c0f8e1086289a63b5fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:58:13.064298 8455 raft_consensus.cc:1261] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750 [term 1 FOLLOWER]: Refusing update from remote peer a5512aa690674c0f8e1086289a63b5fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:58:13.065315 9021 consensus_queue.cc:1035] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c5a71873e04b238c87c1c6773d2750" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:58:13.065692 9015 consensus_queue.cc:1035] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1db5c4ab0241f8bb63d5cce877b6d2" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:58:13.077236 9025 mvcc.cc:205] Tried to move back new op lower bound from 7316250185981313024 to 7316250185802973184. Current Snapshot: MvccSnapshot[applied={T|T < 7316250185981313024}]
I20260808 12:58:13.080704 9027 mvcc.cc:205] Tried to move back new op lower bound from 7316250185981313024 to 7316250185802973184. Current Snapshot: MvccSnapshot[applied={T|T < 7316250185981313024}]
I20260808 12:58:13.095220 8223 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:58:13.095551 8223 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:58:13.099279 8223 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b02d9cb7253e49d1be65f2a4ee5bddc1: Sending DeleteTablet for 3 replicas of tablet ca8f318a70a441dbb41baf282e013b5d
I20260808 12:58:13.100402 8419 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8f318a70a441dbb41baf282e013b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:58:13 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:55764
I20260808 12:58:13.100461 8640 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8f318a70a441dbb41baf282e013b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:58:13 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:45388
I20260808 12:58:13.100962 8859 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8f318a70a441dbb41baf282e013b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:58:13 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:42460
I20260808 12:58:13.102754 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 8320
I20260808 12:58:13.112718 9045 tablet_replica.cc:330] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2: stopping tablet replica
I20260808 12:58:13.113240 9045 raft_consensus.cc:2229] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:58:13.113867 9045 raft_consensus.cc:2258] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:58:13.116459 9045 ts_tablet_manager.cc:1905] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:58:13.127112 9043 tablet_replica.cc:330] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd: stopping tablet replica
I20260808 12:58:13.127626 9043 raft_consensus.cc:2229] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:58:13.128361 9043 raft_consensus.cc:2258] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:58:13.130877 9042 tablet_replica.cc:330] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750: stopping tablet replica
I20260808 12:58:13.131203 9045 ts_tablet_manager.cc:1918] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:58:13.131312 9042 raft_consensus.cc:2229] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:58:13.131543 9045 log.cc:1196] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/ca8f318a70a441dbb41baf282e013b5d
I20260808 12:58:13.131846 9042 raft_consensus.cc:2258] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:58:13.132313 9045 ts_tablet_manager.cc:1939] T ca8f318a70a441dbb41baf282e013b5d P 4c1db5c4ab0241f8bb63d5cce877b6d2: Deleting consensus metadata
I20260808 12:58:13.132766 9043 ts_tablet_manager.cc:1905] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:58:13.135160 8197 catalog_manager.cc:4841] TS 4c1db5c4ab0241f8bb63d5cce877b6d2 (127.24.167.3:39931): tablet ca8f318a70a441dbb41baf282e013b5d (table test-table [id=b5512813a685435f8f18a9026898ff2a]) successfully deleted
Aug 08 12:58:13 dist-test-slave-tk9z krb5kdc[8147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:58:13.140738 9042 ts_tablet_manager.cc:1905] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:58:13.141491 9043 ts_tablet_manager.cc:1918] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:58:13.141695 9043 log.cc:1196] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/ca8f318a70a441dbb41baf282e013b5d
I20260808 12:58:13.142259 9043 ts_tablet_manager.cc:1939] T ca8f318a70a441dbb41baf282e013b5d P a5512aa690674c0f8e1086289a63b5fd: Deleting consensus metadata
I20260808 12:58:13.144299 8199 catalog_manager.cc:4841] TS a5512aa690674c0f8e1086289a63b5fd (127.24.167.2:37333): tablet ca8f318a70a441dbb41baf282e013b5d (table test-table [id=b5512813a685435f8f18a9026898ff2a]) successfully deleted
I20260808 12:58:13.147219 8992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:13.104776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.24.167.1:37595)
0808 12:58:13.105017 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:13.105026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:13.117732 (+ 12706us) server_negotiation.cc:407] Connection header received
0808 12:58:13.126607 (+ 8875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:13.126616 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:13.126739 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:13.126849 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:13.128784 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.130030 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.131943 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.132580 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.132678 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:13.139015 (+ 6337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:13.139026 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:13.139042 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:58:13.139086 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:13.143255 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:13.144305 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:13.144313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:13.144316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:13.144560 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:13.145076 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:13.145082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:13.145085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:13.145604 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:13.146002 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:13.146230 (+ 228us) server_negotiation.cc:299] Negotiation successful
0808 12:58:13.146536 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 12:58:13.147989 8515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:58:13.236871 9042 ts_tablet_manager.cc:1918] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:58:13.237185 9042 log.cc:1196] T ca8f318a70a441dbb41baf282e013b5d P 75c5a71873e04b238c87c1c6773d2750: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/ca8f318a70a441dbb41baf282e013b5d
I20260808 12:58:13.237468 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 8320
W20260808 12:58:13.263108 8199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.167.1:37595 (error 108)
I20260808 12:58:13.263898 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 8539
I20260808 12:58:13.265745 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:13.264685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37595 (local address 127.0.0.1:55786)
0808 12:58:13.265352 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:13.265514 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":457,"threads_started":1}
W20260808 12:58:13.266115 8199 catalog_manager.cc:4584] TS 75c5a71873e04b238c87c1c6773d2750 (127.24.167.1:37595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca8f318a70a441dbb41baf282e013b5d: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37595: connect: Connection refused (error 111)
Aug 08 12:58:13 dist-test-slave-tk9z krb5kdc[8147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 12:58:13.281891 9000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:13.264760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.24.167.2:37333)
0808 12:58:13.264995 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:13.265005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:13.265093 (+ 88us) server_negotiation.cc:407] Connection header received
0808 12:58:13.265427 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:13.265439 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:13.265532 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:13.265655 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:13.267384 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.268211 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.270050 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.270680 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.270777 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:13.275795 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:13.275809 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:13.275825 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:58:13.275869 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:13.278995 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:13.279698 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:13.279711 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:13.279717 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:13.279892 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:13.280280 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:13.280286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:13.280289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:13.280643 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:13.281000 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:13.281303 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 12:58:13.281528 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260808 12:58:13.282655 8737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:58:13.320056 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:13.319495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37595 (local address 127.0.0.1:55800)
0808 12:58:13.319748 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:13.319875 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 12:58:13.321209 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:13.320796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37595 (local address 127.0.0.1:55802)
0808 12:58:13.320955 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:13.321043 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 12:58:13.381807 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:13.381230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37595 (local address 127.0.0.1:55806)
0808 12:58:13.381475 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:13.381615 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 12:58:13.383054 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:13.382625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37595 (local address 127.0.0.1:55822)
0808 12:58:13.382790 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:13.382877 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 12:58:13.398744 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 8539
I20260808 12:58:13.424718 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 8754
Aug 08 12:58:13 dist-test-slave-tk9z krb5kdc[8147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 12:58:13.440980 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:13.425548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.24.167.3:39931)
0808 12:58:13.425748 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 12:58:13.425756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:58:13.425778 (+ 22us) server_negotiation.cc:407] Connection header received
0808 12:58:13.426023 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:58:13.426028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:58:13.426097 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:58:13.426210 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:58:13.427683 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.428822 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.430139 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:58:13.430635 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:58:13.430772 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:58:13.435173 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:58:13.435189 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:58:13.435207 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:58:13.435247 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:58:13.438001 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:13.438797 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:13.438813 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:13.438815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:13.438955 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:58:13.439364 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:58:13.439368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:58:13.439370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:58:13.439746 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:58:13.440076 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0808 12:58:13.440426 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 12:58:13.440651 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 12:58:13.441669 8964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:58:13.464010 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:13.463489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37595 (local address 127.0.0.1:55836)
0808 12:58:13.463726 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:13.463825 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260808 12:58:13.465145 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:13.464708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37595 (local address 127.0.0.1:55838)
0808 12:58:13.464922 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:13.465011 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260808 12:58:13.533046 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 8754
I20260808 12:58:13.563663 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 8171
I20260808 12:58:13.565128 8271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:58:13.681713 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:58:13.679834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37595 (local address 127.0.0.1:55854)
0808 12:58:13.681121 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0808 12:58:13.681505 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":891}
I20260808 12:58:13.681895 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 8171
2026-08-08T12:58:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4911 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:58:13 dist-test-slave-tk9z krb5kdc[9057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:58:13 dist-test-slave-tk9z krb5kdc[9057](info): set up 2 sockets
Aug 08 12:58:13 dist-test-slave-tk9z krb5kdc[9057](info): commencing operation
krb5kdc: starting...
W20260808 12:58:15.002853 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:58:15 dist-test-slave-tk9z krb5kdc[9057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:58:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:58:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.167.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.167.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/rangeradmin_127.24.167.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/rangeradmin_127.24.167.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.167.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.167.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/rangerlookup_127.24.167.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/rangerlookup_127.24.167.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.167.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.167.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/HTTP_127.24.167.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/HTTP_127.24.167.20@KRBTEST.COM.keytab.
I20260808 12:58:15.399704 25244 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMF3Yup/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/postgres -l logfile start
2026-08-08 12:58:17.908 UTC [9102] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 12:58:17.908 UTC [9102] LOG: listening on IPv4 address "127.24.167.20", port 57021
2026-08-08 12:58:17.911 UTC [9102] LOG: listening on Unix socket "/tmp/.s.PGSQL.57021"
2026-08-08 12:58:17.939 UTC [9104] LOG: database system was shut down at 2026-08-08 12:58:16 UTC
2026-08-08 12:58:17.942 UTC [9102] LOG: database system is ready to accept connections
I20260808 12:58:19.144456 25244 mini_postgres.cc:96] Postgres bound to 57021
2026-08-08 12:58:19.169 UTC [9122] FATAL: database "slave" does not exist
127.24.167.20:57021 - accepting connections
I20260808 12:58:19.170615 25244 mini_ranger.cc:161] Starting Ranger...
I20260808 12:58:19.205494 25244 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 12:58:19.678232 25244 mini_ranger.cc:87] Created ranger Postgres database
I20260808 12:58:19.678499 25244 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-admin
2026-08-08 12:58:20,103 [I] DB FLAVOR :POSTGRES
2026-08-08 12:58:20,104 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 12:58:20,104 [I] Checking connection..
2026-08-08 12:58:20,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 12:58:20,360 [I] Checking connection passed.
2026-08-08 12:58:20,360 [I] --------- Verifying version history table ---------
2026-08-08 12:58:20,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 12:58:20,628 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 12:58:20,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 12:58:20,872 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 12:58:20,873 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 12:58:20,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 12:58:21.142 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 12:58:21,152 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 12:58:21,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 12:58:21,398 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 12:58:21,398 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 12:58:21,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 12:58:21,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 12:58:21,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tk9z.c.gcp-upstream.internal','N') ;"
2026-08-08 12:58:22,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 12:58:22,454 [I] Table x_portal_user does not exist in database ranger
2026-08-08 12:58:22,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 12:58:22,732 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 12:58:22,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 12:58:22,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 12:58:23,262 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 12:58:23,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 12:58:24.194 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:24.209 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:24.225 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:24.241 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:24.256 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:24.275 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:24.338 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:24.344 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:24.358 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:24.367 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:24.376 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:24.389 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:24.398 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:24.405 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:24.412 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:24.417 UTC [9412] WARNING: there is no transaction in progress
2026-08-08 12:58:25,009 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 12:58:25,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 12:58:25,261 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 12:58:25,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 12:58:25,540 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 12:58:25,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 12:58:25,814 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 12:58:25,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 12:58:26,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 12:58:26,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tk9z.c.gcp-upstream.internal';"
2026-08-08 12:58:26,593 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 12:58:26,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 12:58:26,830 [I] DB_PATCHES have already been applied
I20260808 12:58:26.836675 25244 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMF3Yup/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 12:58:26.836790 25244 mini_ranger.cc:193] Using host: 127.24.167.20
I20260808 12:58:26.857486 25244 mini_ranger.cc:227] Ranger admin URL: http://127.24.167.20:51631
Aug 08, 2026 12:58:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 12:58:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 12:58:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 12:58:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 12:58:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 12:58:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 12:58:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 12:58:28.325179 25244 mini_ranger.cc:160] Time spent starting Ranger: real 9.155s user 0.003s sys 0.057s
Aug 08, 2026 12:58:49 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 12:58:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 12:58:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 12:58:49 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 12:58:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 12:58:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 12:58:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 12:58:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 12:58:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 12:58:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 12:58:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 12:58:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 12:58:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 12:58:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 12:58:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 12:58:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 12:58:49 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 12:58:50 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 12:58:51.198125 25244 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.167.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.167.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/rangerkms_127.24.167.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/rangerkms_127.24.167.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.167.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.167.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.167.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/HTTP_127.24.167.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.167.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/HTTP_127.24.167.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 12:58:51 dist-test-slave-tk9z krb5kdc[9057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193931, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 12:58:51.416500 25244 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 12:58:51.443171 25244 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 12:58:52.283272 25244 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 12:58:52.283392 25244 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-kms
I20260808 12:58:52.283595 25244 mini_ranger_kms.cc:227] Using postgres at 127.24.167.20:57021
2026-08-08 12:58:52,927 [I] DB FLAVOR :POSTGRES
2026-08-08 12:58:52,927 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 12:58:52,928 [I] Checking connection
2026-08-08 12:58:52,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 12:58:53,170 [I] connection success
2026-08-08 12:58:53,170 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 12:58:53,170 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 12:58:53,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 12:58:53,426 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 12:58:53,427 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 12:58:53,427 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 12:58:53,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:57021/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 12:58:53,684 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 12:58:54.197438 25244 mini_ranger_kms.cc:310] Created kms service
I20260808 12:58:54.396469 25244 mini_ranger_kms.cc:326] Created kudu user
I20260808 12:58:54.444850 25244 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 12:58:54 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193931, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:58:54.873087 25244 mini_ranger_kms.cc:384] Added ranger policy
I20260808 12:58:54.873525 25244 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 12:58:54.873605 25244 mini_ranger_kms.cc:240] Using host: 127.24.167.20
I20260808 12:58:54.898363 25244 mini_ranger_kms.cc:276] Ranger KMS PID: 9723
I20260808 12:58:54.898619 25244 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.167.20:39243
Aug 08, 2026 12:58:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 12:58:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 12:58:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 12:58:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 12:58:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 12:58:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 12:58:57 dist-test-slave-tk9z krb5kdc[9057](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786193937, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.167.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:58:57.613225 25244 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.197s user 0.137s sys 0.484s
I20260808 12:58:57.613585 25244 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 12:58:57.613827 25244 mini_ranger_kms.cc:401] 127.24.167.20:39243/kms/v1/keys
Aug 08 12:58:57 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193937, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.167.20@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
Aug 08 12:58:58 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193937, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.167.20@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
Aug 08 12:59:00 dist-test-slave-tk9z krb5kdc[9057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 12:59:00.722839 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:42755
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:42755
--ranger_config_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.167.20:39243/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:59:01.224689 9807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:59:01.225140 9807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:59:01.225342 9807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:59:01.235486 9807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:59:01.235677 9807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:59:01.235793 9807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:59:01.235950 9807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:59:01.236078 9807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:59:01.249189 9807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.167.20:39243/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:42755
--ranger_config_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:42755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:59:01.250823 9807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:59:01.252363 9807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:59:01.268868 9818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:01.269821 9819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:01.269918 9821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:59:01.270460 9807 server_base.cc:1034] running on GCE node
I20260808 12:59:01.271299 9807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:59:01.272926 9807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:59:01.274212 9807 hybrid_clock.cc:648] HybridClock initialized: now 1786193941274105 us; error 50 us; skew 500 ppm
I20260808 12:59:01.274727 9807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:59:01 dist-test-slave-tk9z krb5kdc[9057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193941, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:59:01.278856 9807 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:59:01.281147 9807 webserver.cc:460] Webserver started at http://127.24.167.62:44497/ using document root <none> and password file <none>
I20260808 12:59:01.282100 9807 fs_manager.cc:359] Metadata directory not provided
I20260808 12:59:01.282337 9807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:59:01.282733 9807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 12:59:01 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193941, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:59:01.347494 9807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "110b11c89ef545caa3deba7161fa9833"
format_stamp: "Formatted at 2026-08-08 12:59:01 on dist-test-slave-tk9z"
server_key: "6b8996a7999070d6417255063c5c2299"
server_key_iv: "09c93c84da67ba12fab0e71f61865624"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:01.348378 9807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "110b11c89ef545caa3deba7161fa9833"
format_stamp: "Formatted at 2026-08-08 12:59:01 on dist-test-slave-tk9z"
server_key: "6b8996a7999070d6417255063c5c2299"
server_key_iv: "09c93c84da67ba12fab0e71f61865624"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:01.355069 9807 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260808 12:59:01.375643 9829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:01.377246 9807 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260808 12:59:01.377454 9807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "110b11c89ef545caa3deba7161fa9833"
format_stamp: "Formatted at 2026-08-08 12:59:01 on dist-test-slave-tk9z"
server_key: "6b8996a7999070d6417255063c5c2299"
server_key_iv: "09c93c84da67ba12fab0e71f61865624"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:01.377666 9807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:59:01.403129 9807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:59:01.408226 9807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:59:01.408711 9807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:59:01.441438 9807 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:42755
I20260808 12:59:01.441565 9917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:42755 every 8 connection(s)
I20260808 12:59:01.443147 9807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:59:01.445225 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 9807
I20260808 12:59:01.445436 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:59:01.446560 25244 external_mini_cluster.cc:1428] Setting key 41a3bc8db3ba5afc6b587f2c167608b3
I20260808 12:59:01.456547 9919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 12:59:01 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:59:01.473001 9919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833: Bootstrap starting.
I20260808 12:59:01.478118 9919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833: Neither blocks nor log segments found. Creating new log.
I20260808 12:59:01.479959 9919 log.cc:824] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833: Log is configured to *not* fsync() on all Append() calls
I20260808 12:59:01.483871 9919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833: No bootstrap required, opened a new log
I20260808 12:59:01.485005 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:01.449132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.24.167.62:42755)
0808 12:59:01.454746 (+ 5614us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:01.454766 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:01.455002 (+ 236us) server_negotiation.cc:407] Connection header received
0808 12:59:01.458416 (+ 3414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:01.458446 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:01.459718 (+ 1272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:01.460237 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:01.461909 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:01.464817 (+ 2908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:01.466642 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:01.467877 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:01.468109 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:01.473512 (+ 5403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:01.473526 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:01.473546 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:01.473592 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:01.478524 (+ 4932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:01.479638 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:01.479647 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:01.479655 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:01.479953 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:01.480437 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:01.480442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:01.480445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:01.481382 (+ 937us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:01.481807 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:01.482532 (+ 725us) server_negotiation.cc:299] Negotiation successful
0808 12:59:01.483337 (+ 805us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4892,"thread_start_us":4538,"threads_started":1}
I20260808 12:59:01.495316 9919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b11c89ef545caa3deba7161fa9833" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42755 } }
I20260808 12:59:01.496002 9919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:59:01.496186 9919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110b11c89ef545caa3deba7161fa9833, State: Initialized, Role: FOLLOWER
I20260808 12:59:01.496861 9919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b11c89ef545caa3deba7161fa9833" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42755 } }
I20260808 12:59:01.497219 9919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:59:01.497473 9919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:59:01.497742 9919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:59:01.501159 9919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b11c89ef545caa3deba7161fa9833" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42755 } }
I20260808 12:59:01.501816 9919 leader_election.cc:302] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 110b11c89ef545caa3deba7161fa9833; no voters:
I20260808 12:59:01.503221 9919 leader_election.cc:288] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:59:01.503425 9924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:59:01.506877 9919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:59:01.509738 9924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [term 1 LEADER]: Becoming Leader. State: Replica: 110b11c89ef545caa3deba7161fa9833, State: Running, Role: LEADER
I20260808 12:59:01.511040 9924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b11c89ef545caa3deba7161fa9833" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42755 } }
I20260808 12:59:01.514912 9925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "110b11c89ef545caa3deba7161fa9833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b11c89ef545caa3deba7161fa9833" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42755 } } }
I20260808 12:59:01.515594 9925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [sys.catalog]: This master's current role is: LEADER
I20260808 12:59:01.527426 9927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 110b11c89ef545caa3deba7161fa9833. Latest consensus state: current_term: 1 leader_uuid: "110b11c89ef545caa3deba7161fa9833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110b11c89ef545caa3deba7161fa9833" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42755 } } }
I20260808 12:59:01.527936 9927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833 [sys.catalog]: This master's current role is: LEADER
I20260808 12:59:01.528844 9941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:59:01.538755 9941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:59:01.560007 9919 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 12:59:01.563081 9941 catalog_manager.cc:1347] Generated new cluster ID: 83672b2e8f8347c48a4b56c415119bd4
I20260808 12:59:01.563230 9941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:59:01.593676 9941 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:59:01.596221 9941 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:59:01.612201 9941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833: Generated new TSK 0
I20260808 12:59:01.614336 9941 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 12:59:02.552971 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.167.20:39243/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42755
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 12:59:02 dist-test-slave-tk9z krb5kdc[9057](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786193942, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 12:59:03.162520 9981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:59:03.162916 9981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:59:03.163153 9981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:59:03.171650 9981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:59:03.171855 9981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:59:03.172077 9981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:59:03.189103 9981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.167.20:39243/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:59:03.191788 9981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:59:03.194056 9981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:59:03.219425 9996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:03.219704 9995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:03.227658 10003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:59:03.228744 9981 server_base.cc:1034] running on GCE node
I20260808 12:59:03.229615 9981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:59:03.231354 9981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:59:03.232666 9981 hybrid_clock.cc:648] HybridClock initialized: now 1786193943232599 us; error 54 us; skew 500 ppm
I20260808 12:59:03.233232 9981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:59:03 dist-test-slave-tk9z krb5kdc[9057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193943, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:59:03.239475 9981 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 12:59:03.242544 9981 webserver.cc:460] Webserver started at http://127.24.167.1:40971/ using document root <none> and password file <none>
I20260808 12:59:03.243901 9981 fs_manager.cc:359] Metadata directory not provided
I20260808 12:59:03.244200 9981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:59:03.245002 9981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 12:59:03 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193943, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:59:03.271524 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "c92ff09f5c344d76bca2aa685faa7225"
format_stamp: "Formatted at 2026-08-08 12:59:03 on dist-test-slave-tk9z"
server_key: "15b43d670242471a3b64cf526552bc5e"
server_key_iv: "91181eb11e08d00a63190461cbd8a8eb"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:03.273396 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "c92ff09f5c344d76bca2aa685faa7225"
format_stamp: "Formatted at 2026-08-08 12:59:03 on dist-test-slave-tk9z"
server_key: "15b43d670242471a3b64cf526552bc5e"
server_key_iv: "91181eb11e08d00a63190461cbd8a8eb"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:03.290448 9981 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.007s sys 0.007s
I20260808 12:59:03.326843 10012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:03.329272 9981 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.008s
I20260808 12:59:03.329510 9981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "c92ff09f5c344d76bca2aa685faa7225"
format_stamp: "Formatted at 2026-08-08 12:59:03 on dist-test-slave-tk9z"
server_key: "15b43d670242471a3b64cf526552bc5e"
server_key_iv: "91181eb11e08d00a63190461cbd8a8eb"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:03.329846 9981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:59:03.372579 9981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:59:03.378623 9981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:59:03.379088 9981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:59:03.382256 9981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:59:03.384887 9981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:59:03.385036 9981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:03.385202 9981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:59:03.385342 9981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 12:59:03 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193942, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:59:03.666666 9981 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:41381
I20260808 12:59:03.666755 10183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:41381 every 8 connection(s)
I20260808 12:59:03.669221 9981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:59:03.674950 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 9981
I20260808 12:59:03.675173 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:59:03.676201 25244 external_mini_cluster.cc:1428] Setting key 3f9e174d28686d30114ee5784f789674
Aug 08 12:59:03 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193943, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
I20260808 12:59:03.727288 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:03.673661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:32809 (local address 127.24.167.62:42755)
0808 12:59:03.689231 (+ 15570us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:03.689238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:03.696330 (+ 7092us) server_negotiation.cc:407] Connection header received
0808 12:59:03.700425 (+ 4095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:03.700433 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:03.700535 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:03.700626 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:03.703786 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:03.705255 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:03.708890 (+ 3635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:03.709759 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:03.709867 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:03.716794 (+ 6927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:03.716802 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:03.716809 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:03.716884 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:03.722785 (+ 5901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:03.723906 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:03.723912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:03.723915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:03.724121 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:03.724851 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:03.724856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:03.724859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:03.725240 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:03.725405 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:03.726490 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0808 12:59:03.726742 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15358,"thread_start_us":15535,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
I20260808 12:59:03.730624 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:03.677823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42755 (local address 127.24.167.1:32809)
0808 12:59:03.696069 (+ 18246us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:03.696147 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:03.699993 (+ 3846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:03.701334 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:03.701348 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:03.702277 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:59:03.703537 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:03.703568 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:03.705532 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:03.705538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:03.708705 (+ 3167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:03.708722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:03.710398 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:03.710410 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:03.710799 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:03.711908 (+ 1109us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:59:03.711986 (+ 78us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:59:03.716420 (+ 4434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:59:03.723011 (+ 6591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:03.723022 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:03.723044 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:03.723746 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:03.724361 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:03.724366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:03.724370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:03.724702 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:03.725508 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:59:03.725516 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:59:03.725988 (+ 472us) client_negotiation.cc:769] Sending connection context
0808 12:59:03.727216 (+ 1228us) client_negotiation.cc:240] Negotiation successful
0808 12:59:03.728250 (+ 1034us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17392,"thread_start_us":17023,"threads_started":1}
I20260808 12:59:03.733196 10184 heartbeater.cc:347] Connected to a master server at 127.24.167.62:42755
I20260808 12:59:03.733690 10184 heartbeater.cc:464] Registering TS with master...
I20260808 12:59:03.735255 10184 heartbeater.cc:511] Master 127.24.167.62:42755 requested a full tablet report, sending...
I20260808 12:59:03.738355 9855 ts_manager.cc:194] Registered new tserver with Master: c92ff09f5c344d76bca2aa685faa7225 (127.24.167.1:41381)
I20260808 12:59:03.740741 9855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:32809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 12:59:03.774349 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.167.20:39243/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42755
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 12:59:03 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193942, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
W20260808 12:59:04.477381 10191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:59:04.477723 10191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:59:04.477975 10191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:59:04.487071 10191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:59:04.487318 10191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:59:04.487529 10191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:59:04.500097 10191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.167.20:39243/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:59:04.501524 10191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:59:04.503163 10191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:59:04.521147 10222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:04.521080 10219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:04.521445 10220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:59:04.523011 10191 server_base.cc:1034] running on GCE node
I20260808 12:59:04.523658 10191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:59:04.524655 10191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:59:04.525882 10191 hybrid_clock.cc:648] HybridClock initialized: now 1786193944525821 us; error 47 us; skew 500 ppm
I20260808 12:59:04.526244 10191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:59:04 dist-test-slave-tk9z krb5kdc[9057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193944, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:59:04.529717 10191 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 12:59:04.531683 10191 webserver.cc:460] Webserver started at http://127.24.167.2:43137/ using document root <none> and password file <none>
I20260808 12:59:04.532408 10191 fs_manager.cc:359] Metadata directory not provided
I20260808 12:59:04.532573 10191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:59:04.532909 10191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 12:59:04 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193944, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:59:04.550235 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "919748d486164effa6c4a302522aa258"
format_stamp: "Formatted at 2026-08-08 12:59:04 on dist-test-slave-tk9z"
server_key: "b942d9e3419915bec9e60f8e5349ced6"
server_key_iv: "79331d5d5ac7887f34f493c224ad4f62"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:04.551469 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "919748d486164effa6c4a302522aa258"
format_stamp: "Formatted at 2026-08-08 12:59:04 on dist-test-slave-tk9z"
server_key: "b942d9e3419915bec9e60f8e5349ced6"
server_key_iv: "79331d5d5ac7887f34f493c224ad4f62"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:04.559247 10191 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260808 12:59:04.576555 10233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:04.578265 10191 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260808 12:59:04.578572 10191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "919748d486164effa6c4a302522aa258"
format_stamp: "Formatted at 2026-08-08 12:59:04 on dist-test-slave-tk9z"
server_key: "b942d9e3419915bec9e60f8e5349ced6"
server_key_iv: "79331d5d5ac7887f34f493c224ad4f62"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:04.578908 10191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:59:04.607127 10191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:59:04.612661 10191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:59:04.613245 10191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:59:04.614813 10191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:59:04.616912 10191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:59:04.617090 10191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:04.617405 10191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:59:04.617553 10191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:04.703411 10191 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:44809
I20260808 12:59:04.703599 10420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:44809 every 8 connection(s)
I20260808 12:59:04.704919 10191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:59:04.709270 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 10191
I20260808 12:59:04.709501 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:59:04.710224 25244 external_mini_cluster.cc:1428] Setting key 9368f3c96bb33f94e3cc25a47963e4fc
Aug 08 12:59:04 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193944, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:59:04.746150 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:04.708965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:50065 (local address 127.24.167.62:42755)
0808 12:59:04.724861 (+ 15896us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:04.724869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:04.724898 (+ 29us) server_negotiation.cc:407] Connection header received
0808 12:59:04.726737 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:04.726747 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:04.726847 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:04.726969 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:04.729794 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:04.730933 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:04.733219 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:04.733703 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:04.733856 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:04.738659 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:04.738664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:04.738667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:04.738733 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:04.742185 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:04.743147 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:04.743153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:04.743158 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:04.743354 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:04.743868 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:04.743877 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:04.743879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:04.744269 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:04.744527 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:04.745408 (+ 881us) server_negotiation.cc:299] Negotiation successful
0808 12:59:04.745667 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15731,"thread_start_us":15913,"threads_started":1}
I20260808 12:59:04.747430 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:04.709540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42755 (local address 127.24.167.2:50065)
0808 12:59:04.723518 (+ 13978us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:04.723583 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:04.726294 (+ 2711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:04.727410 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:04.727424 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:04.728441 (+ 1017us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:59:04.729577 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:04.729607 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:04.731107 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:04.731112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:04.733062 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:04.733076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:04.734062 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:04.734072 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:04.734429 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:04.735351 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:59:04.735376 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:59:04.738345 (+ 2969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:59:04.742431 (+ 4086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:04.742441 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:04.742463 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:04.743007 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:04.743481 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:04.743486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:04.743490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:04.743730 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:04.744433 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:59:04.744442 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:59:04.744900 (+ 458us) client_negotiation.cc:769] Sending connection context
0808 12:59:04.745201 (+ 301us) client_negotiation.cc:240] Negotiation successful
0808 12:59:04.745907 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13606,"thread_start_us":13225,"threads_started":1}
I20260808 12:59:04.748529 10184 heartbeater.cc:503] Master 127.24.167.62:42755 was elected leader, sending a full tablet report...
I20260808 12:59:04.749852 10421 heartbeater.cc:347] Connected to a master server at 127.24.167.62:42755
I20260808 12:59:04.750204 10421 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
I20260808 12:59:04.751300 10421 heartbeater.cc:511] Master 127.24.167.62:42755 requested a full tablet report, sending...
I20260808 12:59:04.753047 9855 ts_manager.cc:194] Registered new tserver with Master: 919748d486164effa6c4a302522aa258 (127.24.167.2:44809)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
I20260808 12:59:04.754145 9855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:50065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 12:59:04.796036 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.167.20:39243/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42755
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:59:05.356382 10433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:59:05.356755 10433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:59:05.356863 10433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:59:05.366707 10433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:59:05.366889 10433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:59:05.367066 10433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:59:05.379436 10433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.167.20:39243/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:59:05.380937 10433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:59:05.382504 10433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:59:05.400300 10440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:05.401206 10441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:05.401201 10444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:59:05.403646 10433 server_base.cc:1034] running on GCE node
I20260808 12:59:05.404479 10433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:59:05.405802 10433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:59:05.407138 10433 hybrid_clock.cc:648] HybridClock initialized: now 1786193945407052 us; error 79 us; skew 500 ppm
I20260808 12:59:05.407729 10433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:59:05 dist-test-slave-tk9z krb5kdc[9057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193945, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:59:05.418097 10433 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 12:59:05.420676 10433 webserver.cc:460] Webserver started at http://127.24.167.3:43443/ using document root <none> and password file <none>
I20260808 12:59:05.421752 10433 fs_manager.cc:359] Metadata directory not provided
I20260808 12:59:05.421959 10433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:59:05.422498 10433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 12:59:05 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193945, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:59:05.442644 10433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "043cff765f7d494a8a1386cc3451f2ba"
format_stamp: "Formatted at 2026-08-08 12:59:05 on dist-test-slave-tk9z"
server_key: "70586483504c5f2af54f9e09cfbd2ef8"
server_key_iv: "73d1377ca6356d39d050ba10349449f3"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:05.444120 10433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "043cff765f7d494a8a1386cc3451f2ba"
format_stamp: "Formatted at 2026-08-08 12:59:05 on dist-test-slave-tk9z"
server_key: "70586483504c5f2af54f9e09cfbd2ef8"
server_key_iv: "73d1377ca6356d39d050ba10349449f3"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:05.453697 10433 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.007s
I20260808 12:59:05.473397 10456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:05.475453 10433 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260808 12:59:05.475711 10433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "043cff765f7d494a8a1386cc3451f2ba"
format_stamp: "Formatted at 2026-08-08 12:59:05 on dist-test-slave-tk9z"
server_key: "70586483504c5f2af54f9e09cfbd2ef8"
server_key_iv: "73d1377ca6356d39d050ba10349449f3"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:05.476099 10433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:59:05.497241 10433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:59:05.502511 10433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:59:05.502904 10433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:59:05.504278 10433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:59:05.506291 10433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:59:05.506464 10433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:05.506673 10433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:59:05.506767 10433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:05.579865 10433 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:45831
I20260808 12:59:05.580024 10646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:45831 every 8 connection(s)
I20260808 12:59:05.581533 10433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:59:05.584498 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 10433
I20260808 12:59:05.584720 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:59:05.585326 25244 external_mini_cluster.cc:1428] Setting key 5a724ea97a667500df65b423e59704d2
Aug 08 12:59:05 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193945, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:59:05.608124 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:05.585098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:53449 (local address 127.24.167.62:42755)
0808 12:59:05.585904 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:05.585908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:05.587627 (+ 1719us) server_negotiation.cc:407] Connection header received
0808 12:59:05.590964 (+ 3337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:05.590968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:05.591045 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:05.591136 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:05.593401 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.594340 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.596405 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.596976 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.597099 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:05.601457 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:05.601464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:05.601470 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:05.601537 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:05.605092 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:05.605826 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:05.605830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:05.605833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:05.605958 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:05.606423 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:05.606428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:05.606432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:05.606784 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:05.606929 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:05.607634 (+ 705us) server_negotiation.cc:299] Negotiation successful
0808 12:59:05.607815 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":516,"threads_started":1}
I20260808 12:59:05.609351 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:05.585948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42755 (local address 127.24.167.3:53449)
0808 12:59:05.587444 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:05.587501 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:05.590655 (+ 3154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:05.591518 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:05.591535 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:05.592402 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:59:05.593232 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:05.593252 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.594595 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.594600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:05.596271 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:05.596285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.597212 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.597218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:05.597505 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:05.598240 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:59:05.598262 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:59:05.601115 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:59:05.605218 (+ 4103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:05.605225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:05.605236 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:05.605703 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:05.606069 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:05.606072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:05.606074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:05.606276 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:05.606906 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:59:05.606912 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:59:05.607279 (+ 367us) client_negotiation.cc:769] Sending connection context
0808 12:59:05.607553 (+ 274us) client_negotiation.cc:240] Negotiation successful
0808 12:59:05.607971 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"thread_start_us":839,"threads_started":1}
I20260808 12:59:05.610899 10649 heartbeater.cc:347] Connected to a master server at 127.24.167.62:42755
I20260808 12:59:05.611229 10649 heartbeater.cc:464] Registering TS with master...
I20260808 12:59:05.612177 10649 heartbeater.cc:511] Master 127.24.167.62:42755 requested a full tablet report, sending...
I20260808 12:59:05.614094 9855 ts_manager.cc:194] Registered new tserver with Master: 043cff765f7d494a8a1386cc3451f2ba (127.24.167.3:45831)
I20260808 12:59:05.615216 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 12:59:05.615414 9855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:53449
I20260808 12:59:05.638002 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:05.621097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38552 (local address 127.24.167.62:42755)
0808 12:59:05.621259 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:05.621263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:05.622018 (+ 755us) server_negotiation.cc:407] Connection header received
0808 12:59:05.622251 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:05.622256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:05.622327 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:05.622539 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:05.624437 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.625367 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.627206 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.627894 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.628037 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:05.631291 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:05.631295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:05.631298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:05.631331 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:05.634661 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:05.635484 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:05.635490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:05.635495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:05.635704 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:05.636149 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:05.636154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:05.636159 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:05.636489 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:05.636844 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:05.637251 (+ 407us) server_negotiation.cc:299] Negotiation successful
0808 12:59:05.637592 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 12:59:05.642922 9855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:59:05.652964 9855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:59:05.700044 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:05.669915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:41381 (local address 127.0.0.1:42174)
0808 12:59:05.685135 (+ 15220us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:05.685229 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:05.685483 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:05.686969 (+ 1486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:05.686977 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:05.687023 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:59:05.689161 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:05.689178 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.692319 (+ 3141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.692324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:05.695751 (+ 3427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:05.695761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.698352 (+ 2591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.698361 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:05.698770 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:05.698838 (+ 68us) client_negotiation.cc:769] Sending connection context
0808 12:59:05.699011 (+ 173us) client_negotiation.cc:240] Negotiation successful
0808 12:59:05.699338 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14768,"mutex_wait_us":99,"spinlock_wait_cycles":1280,"thread_start_us":14438,"threads_started":1}
I20260808 12:59:05.700151 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:05.670869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.24.167.1:41381)
0808 12:59:05.683563 (+ 12694us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:05.683576 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:05.685714 (+ 2138us) server_negotiation.cc:407] Connection header received
0808 12:59:05.685916 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:05.685933 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:05.686285 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:05.686525 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:59:05.689407 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.692104 (+ 2697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.695989 (+ 3885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.698003 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.698155 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:05.698472 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:05.699133 (+ 661us) server_negotiation.cc:299] Negotiation successful
0808 12:59:05.699358 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12366,"thread_start_us":12737,"threads_started":1}
I20260808 12:59:05.700150 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:05.669948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.24.167.2:44809)
0808 12:59:05.685473 (+ 15525us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:05.685482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:05.686673 (+ 1191us) server_negotiation.cc:407] Connection header received
0808 12:59:05.686900 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:05.686912 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:05.687187 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:05.687611 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:59:05.689692 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.692071 (+ 2379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.695928 (+ 3857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.697993 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.698154 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:05.698399 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:05.699260 (+ 861us) server_negotiation.cc:299] Negotiation successful
0808 12:59:05.699429 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15247,"thread_start_us":15463,"threads_started":1}
I20260808 12:59:05.701159 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:05.670411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:44809 (local address 127.0.0.1:43744)
0808 12:59:05.686457 (+ 16046us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:05.686478 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:05.686653 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:05.687517 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:05.687522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:05.687555 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:59:05.689397 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:05.689414 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.692242 (+ 2828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.692248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:05.695716 (+ 3468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:05.695734 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.698759 (+ 3025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.698766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:05.698905 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:05.698930 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 12:59:05.699055 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 12:59:05.700657 (+ 1602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15855,"mutex_wait_us":1393,"thread_start_us":16909,"threads_started":1}
I20260808 12:59:05.702744 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:05.673583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:45831 (local address 127.0.0.1:53512)
0808 12:59:05.688288 (+ 14705us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:05.688310 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:05.688912 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:05.690256 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:05.690260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:05.690300 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:59:05.690960 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:05.690971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.693897 (+ 2926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.693901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:05.697155 (+ 3254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:05.697169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.701543 (+ 4374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.701555 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:05.701784 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:05.701836 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 12:59:05.701927 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 12:59:05.702089 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14479,"mutex_wait_us":2,"thread_start_us":12505,"threads_started":1}
I20260808 12:59:05.704871 10087 tablet_service.cc:1467] Processing CreateTablet for tablet 48e85e8dba3c42e9bb1f4373e469d01b (DEFAULT_TABLE table=test-table [id=6159846038454ec6b392309acf925d6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:59:05.706284 10087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e85e8dba3c42e9bb1f4373e469d01b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:59:05.705844 10310 tablet_service.cc:1467] Processing CreateTablet for tablet 48e85e8dba3c42e9bb1f4373e469d01b (DEFAULT_TABLE table=test-table [id=6159846038454ec6b392309acf925d6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:59:05.707579 10310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e85e8dba3c42e9bb1f4373e469d01b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:59:05.707666 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:05.674875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.24.167.3:45831)
0808 12:59:05.682110 (+ 7235us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:05.682124 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:05.688430 (+ 6306us) server_negotiation.cc:407] Connection header received
0808 12:59:05.689768 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:05.689775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:05.690051 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:05.690282 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:59:05.691254 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.692772 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.697403 (+ 4631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.699324 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.699463 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:05.699794 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:05.706701 (+ 6907us) server_negotiation.cc:299] Negotiation successful
0808 12:59:05.707007 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5942,"thread_start_us":5784,"threads_started":1}
I20260808 12:59:05.712467 10544 tablet_service.cc:1467] Processing CreateTablet for tablet 48e85e8dba3c42e9bb1f4373e469d01b (DEFAULT_TABLE table=test-table [id=6159846038454ec6b392309acf925d6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:59:05.715379 10544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e85e8dba3c42e9bb1f4373e469d01b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:59:05.725122 10686 tablet_bootstrap.cc:492] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258: Bootstrap starting.
I20260808 12:59:05.730521 10686 tablet_bootstrap.cc:654] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258: Neither blocks nor log segments found. Creating new log.
I20260808 12:59:05.732141 10686 log.cc:824] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258: Log is configured to *not* fsync() on all Append() calls
I20260808 12:59:05.735323 10687 tablet_bootstrap.cc:492] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225: Bootstrap starting.
I20260808 12:59:05.736652 10686 tablet_bootstrap.cc:492] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258: No bootstrap required, opened a new log
I20260808 12:59:05.737106 10686 ts_tablet_manager.cc:1397] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260808 12:59:05.739068 10690 tablet_bootstrap.cc:492] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba: Bootstrap starting.
I20260808 12:59:05.740059 10687 tablet_bootstrap.cc:654] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225: Neither blocks nor log segments found. Creating new log.
I20260808 12:59:05.742009 10687 log.cc:824] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225: Log is configured to *not* fsync() on all Append() calls
I20260808 12:59:05.743832 10690 tablet_bootstrap.cc:654] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba: Neither blocks nor log segments found. Creating new log.
I20260808 12:59:05.745141 10690 log.cc:824] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba: Log is configured to *not* fsync() on all Append() calls
I20260808 12:59:05.746091 10687 tablet_bootstrap.cc:492] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225: No bootstrap required, opened a new log
I20260808 12:59:05.746603 10687 ts_tablet_manager.cc:1397] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260808 12:59:05.748988 10690 tablet_bootstrap.cc:492] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba: No bootstrap required, opened a new log
I20260808 12:59:05.749320 10690 ts_tablet_manager.cc:1397] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260808 12:59:05.752310 10686 raft_consensus.cc:348] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919748d486164effa6c4a302522aa258" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44809 } } peers { permanent_uuid: "c92ff09f5c344d76bca2aa685faa7225" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 41381 } } peers { permanent_uuid: "043cff765f7d494a8a1386cc3451f2ba" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45831 } }
I20260808 12:59:05.753194 10686 raft_consensus.cc:374] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:59:05.753409 10686 raft_consensus.cc:729] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 919748d486164effa6c4a302522aa258, State: Initialized, Role: FOLLOWER
I20260808 12:59:05.754585 10686 consensus_queue.cc:260] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919748d486164effa6c4a302522aa258" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44809 } } peers { permanent_uuid: "c92ff09f5c344d76bca2aa685faa7225" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 41381 } } peers { permanent_uuid: "043cff765f7d494a8a1386cc3451f2ba" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45831 } }
I20260808 12:59:05.756490 10421 heartbeater.cc:503] Master 127.24.167.62:42755 was elected leader, sending a full tablet report...
I20260808 12:59:05.757457 10686 ts_tablet_manager.cc:1428] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258: Time spent starting tablet: real 0.020s user 0.019s sys 0.000s
I20260808 12:59:05.761678 10687 raft_consensus.cc:348] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919748d486164effa6c4a302522aa258" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44809 } } peers { permanent_uuid: "c92ff09f5c344d76bca2aa685faa7225" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 41381 } } peers { permanent_uuid: "043cff765f7d494a8a1386cc3451f2ba" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45831 } }
I20260808 12:59:05.761617 10690 raft_consensus.cc:348] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919748d486164effa6c4a302522aa258" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44809 } } peers { permanent_uuid: "c92ff09f5c344d76bca2aa685faa7225" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 41381 } } peers { permanent_uuid: "043cff765f7d494a8a1386cc3451f2ba" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45831 } }
I20260808 12:59:05.762101 10687 raft_consensus.cc:374] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:59:05.762197 10690 raft_consensus.cc:374] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:59:05.762252 10687 raft_consensus.cc:729] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92ff09f5c344d76bca2aa685faa7225, State: Initialized, Role: FOLLOWER
I20260808 12:59:05.762409 10690 raft_consensus.cc:729] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043cff765f7d494a8a1386cc3451f2ba, State: Initialized, Role: FOLLOWER
I20260808 12:59:05.763212 10690 consensus_queue.cc:260] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919748d486164effa6c4a302522aa258" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44809 } } peers { permanent_uuid: "c92ff09f5c344d76bca2aa685faa7225" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 41381 } } peers { permanent_uuid: "043cff765f7d494a8a1386cc3451f2ba" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45831 } }
I20260808 12:59:05.763331 10687 consensus_queue.cc:260] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919748d486164effa6c4a302522aa258" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44809 } } peers { permanent_uuid: "c92ff09f5c344d76bca2aa685faa7225" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 41381 } } peers { permanent_uuid: "043cff765f7d494a8a1386cc3451f2ba" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45831 } }
I20260808 12:59:05.767194 10649 heartbeater.cc:503] Master 127.24.167.62:42755 was elected leader, sending a full tablet report...
I20260808 12:59:05.768193 10690 ts_tablet_manager.cc:1428] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba: Time spent starting tablet: real 0.019s user 0.007s sys 0.012s
I20260808 12:59:05.772109 10687 ts_tablet_manager.cc:1428] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225: Time spent starting tablet: real 0.025s user 0.015s sys 0.009s
W20260808 12:59:05.834227 10650 tablet.cc:2369] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:59:05.881654 10697 raft_consensus.cc:482] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:59:05.882332 10697 raft_consensus.cc:504] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919748d486164effa6c4a302522aa258" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44809 } } peers { permanent_uuid: "c92ff09f5c344d76bca2aa685faa7225" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 41381 } } peers { permanent_uuid: "043cff765f7d494a8a1386cc3451f2ba" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45831 } }
I20260808 12:59:05.885293 10697 leader_election.cc:288] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 919748d486164effa6c4a302522aa258 (127.24.167.2:44809), 043cff765f7d494a8a1386cc3451f2ba (127.24.167.3:45831)
I20260808 12:59:05.895431 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:05.885920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:60627 (local address 127.24.167.2:44809)
0808 12:59:05.886145 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:05.886165 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:05.887712 (+ 1547us) server_negotiation.cc:407] Connection header received
0808 12:59:05.888028 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:05.888032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:05.888100 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:05.888201 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:59:05.889451 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.890352 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.893059 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.894039 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.894263 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:05.894426 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:05.894778 (+ 352us) server_negotiation.cc:299] Negotiation successful
0808 12:59:05.894914 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 12:59:05.895715 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:05.885931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:44809 (local address 127.24.167.1:60627)
0808 12:59:05.887590 (+ 1659us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:05.887610 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:05.887854 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:05.888228 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:05.888233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:05.888298 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:59:05.889284 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:05.889296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.890551 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.890556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:05.892858 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:05.892871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.894403 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.894412 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:05.894602 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:05.894629 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 12:59:05.894749 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 12:59:05.894906 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1477,"mutex_wait_us":58,"spinlock_wait_cycles":15616,"thread_start_us":1029,"threads_started":1}
I20260808 12:59:05.895778 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:05.885681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:51943 (local address 127.24.167.3:45831)
0808 12:59:05.885942 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:05.885949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:05.887740 (+ 1791us) server_negotiation.cc:407] Connection header received
0808 12:59:05.888031 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:05.888037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:05.888113 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:05.888217 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:59:05.889756 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.890767 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.893490 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.894615 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.894843 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:05.894948 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:05.895274 (+ 326us) server_negotiation.cc:299] Negotiation successful
0808 12:59:05.895404 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 12:59:05.896584 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:05.886017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:45831 (local address 127.24.167.1:51943)
0808 12:59:05.887611 (+ 1594us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:05.887627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:05.887867 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:05.888232 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:05.888236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:05.888281 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:59:05.889553 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:05.889564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.890955 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.890958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:05.893316 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:05.893329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.894954 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.894962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:05.895111 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:05.895134 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 12:59:05.895245 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 12:59:05.895373 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1473,"mutex_wait_us":87,"spinlock_wait_cycles":96640,"thread_start_us":933,"threads_started":1}
I20260808 12:59:05.897262 10344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e85e8dba3c42e9bb1f4373e469d01b" candidate_uuid: "c92ff09f5c344d76bca2aa685faa7225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "919748d486164effa6c4a302522aa258" is_pre_election: true
I20260808 12:59:05.898006 10575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e85e8dba3c42e9bb1f4373e469d01b" candidate_uuid: "c92ff09f5c344d76bca2aa685faa7225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "043cff765f7d494a8a1386cc3451f2ba" is_pre_election: true
I20260808 12:59:05.898334 10344 raft_consensus.cc:2454] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92ff09f5c344d76bca2aa685faa7225 in term 0.
I20260808 12:59:05.898764 10575 raft_consensus.cc:2454] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92ff09f5c344d76bca2aa685faa7225 in term 0.
I20260808 12:59:05.899509 10013 leader_election.cc:302] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 919748d486164effa6c4a302522aa258, c92ff09f5c344d76bca2aa685faa7225; no voters:
I20260808 12:59:05.900219 10697 raft_consensus.cc:2781] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:59:05.900480 10697 raft_consensus.cc:482] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:59:05.900628 10697 raft_consensus.cc:3037] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:59:05.904143 10697 raft_consensus.cc:504] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919748d486164effa6c4a302522aa258" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44809 } } peers { permanent_uuid: "c92ff09f5c344d76bca2aa685faa7225" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 41381 } } peers { permanent_uuid: "043cff765f7d494a8a1386cc3451f2ba" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45831 } }
I20260808 12:59:05.905128 10697 leader_election.cc:288] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [CANDIDATE]: Term 1 election: Requested vote from peers 919748d486164effa6c4a302522aa258 (127.24.167.2:44809), 043cff765f7d494a8a1386cc3451f2ba (127.24.167.3:45831)
I20260808 12:59:05.905786 10575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e85e8dba3c42e9bb1f4373e469d01b" candidate_uuid: "c92ff09f5c344d76bca2aa685faa7225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "043cff765f7d494a8a1386cc3451f2ba"
I20260808 12:59:05.905745 10344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e85e8dba3c42e9bb1f4373e469d01b" candidate_uuid: "c92ff09f5c344d76bca2aa685faa7225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "919748d486164effa6c4a302522aa258"
I20260808 12:59:05.906066 10575 raft_consensus.cc:3037] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:59:05.906091 10344 raft_consensus.cc:3037] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:59:05.911211 10344 raft_consensus.cc:2454] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92ff09f5c344d76bca2aa685faa7225 in term 1.
I20260808 12:59:05.911211 10575 raft_consensus.cc:2454] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92ff09f5c344d76bca2aa685faa7225 in term 1.
I20260808 12:59:05.911922 10014 leader_election.cc:302] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 043cff765f7d494a8a1386cc3451f2ba, c92ff09f5c344d76bca2aa685faa7225; no voters:
I20260808 12:59:05.912590 10697 raft_consensus.cc:2781] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:59:05.913379 10697 raft_consensus.cc:686] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [term 1 LEADER]: Becoming Leader. State: Replica: c92ff09f5c344d76bca2aa685faa7225, State: Running, Role: LEADER
I20260808 12:59:05.914043 10697 consensus_queue.cc:237] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919748d486164effa6c4a302522aa258" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44809 } } peers { permanent_uuid: "c92ff09f5c344d76bca2aa685faa7225" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 41381 } } peers { permanent_uuid: "043cff765f7d494a8a1386cc3451f2ba" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45831 } }
I20260808 12:59:05.920840 9854 catalog_manager.cc:5495] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92ff09f5c344d76bca2aa685faa7225 (127.24.167.1). New cstate: current_term: 1 leader_uuid: "c92ff09f5c344d76bca2aa685faa7225" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919748d486164effa6c4a302522aa258" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92ff09f5c344d76bca2aa685faa7225" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 41381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "043cff765f7d494a8a1386cc3451f2ba" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45831 } health_report { overall_health: UNKNOWN } } }
W20260808 12:59:05.943493 10185 tablet.cc:2369] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 12:59:05.958832 10423 tablet.cc:2369] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:59:05.991726 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:05.984737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.24.167.1:41381)
0808 12:59:05.985058 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:05.985063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:05.985083 (+ 20us) server_negotiation.cc:407] Connection header received
0808 12:59:05.985286 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:05.985290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:05.985381 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:05.985510 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:59:05.986029 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.987041 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.989143 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:05.989751 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:05.989852 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:05.990474 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:59:05.991083 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:59:05.991195 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:05.991326 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 12:59:05.991400 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260808 12:59:06.001732 10575 raft_consensus.cc:1261] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba [term 1 FOLLOWER]: Refusing update from remote peer c92ff09f5c344d76bca2aa685faa7225: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:59:06.001864 10344 raft_consensus.cc:1261] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258 [term 1 FOLLOWER]: Refusing update from remote peer c92ff09f5c344d76bca2aa685faa7225: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:59:06.002991 10700 consensus_queue.cc:1035] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "043cff765f7d494a8a1386cc3451f2ba" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:59:06.003731 10697 consensus_queue.cc:1035] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "919748d486164effa6c4a302522aa258" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 44809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:59:06.020069 10707 mvcc.cc:205] Tried to move back new op lower bound from 7316250402811367424 to 7316250402468470784. Current Snapshot: MvccSnapshot[applied={T|T < 7316250402811367424}]
I20260808 12:59:06.036033 9855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:59:06.036441 9855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 12:59:06.041368 9855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 110b11c89ef545caa3deba7161fa9833: Sending DeleteTablet for 3 replicas of tablet 48e85e8dba3c42e9bb1f4373e469d01b
I20260808 12:59:06.042857 10087 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e85e8dba3c42e9bb1f4373e469d01b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:59:06 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:42174
I20260808 12:59:06.042826 10310 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e85e8dba3c42e9bb1f4373e469d01b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:59:06 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:43744
I20260808 12:59:06.042905 10544 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e85e8dba3c42e9bb1f4373e469d01b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 12:59:06 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:53512
I20260808 12:59:06.045970 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 9981
I20260808 12:59:06.055510 10722 tablet_replica.cc:330] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225: stopping tablet replica
I20260808 12:59:06.056071 10722 raft_consensus.cc:2229] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [term 1 LEADER]: Raft consensus shutting down.
I20260808 12:59:06.057189 10724 tablet_replica.cc:330] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba: stopping tablet replica
I20260808 12:59:06.057773 10724 raft_consensus.cc:2229] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:59:06.058027 10723 tablet_replica.cc:330] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258: stopping tablet replica
I20260808 12:59:06.058478 10724 raft_consensus.cc:2258] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:59:06.058531 10723 raft_consensus.cc:2229] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 12:59:06.059073 10723 raft_consensus.cc:2258] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:59:06.059358 10722 raft_consensus.cc:2258] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 12:59:06.061463 10723 ts_tablet_manager.cc:1905] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:59:06.061992 10722 ts_tablet_manager.cc:1905] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 12:59:06 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:59:06.069182 10723 ts_tablet_manager.cc:1918] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:59:06.069465 10723 log.cc:1196] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/48e85e8dba3c42e9bb1f4373e469d01b
I20260808 12:59:06.070185 10723 ts_tablet_manager.cc:1939] T 48e85e8dba3c42e9bb1f4373e469d01b P 919748d486164effa6c4a302522aa258: Deleting consensus metadata
I20260808 12:59:06.071689 10724 ts_tablet_manager.cc:1905] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 12:59:06.072964 9830 catalog_manager.cc:4841] TS 919748d486164effa6c4a302522aa258 (127.24.167.2:44809): tablet 48e85e8dba3c42e9bb1f4373e469d01b (table test-table [id=6159846038454ec6b392309acf925d6e]) successfully deleted
I20260808 12:59:06.073369 10722 ts_tablet_manager.cc:1918] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:59:06.074635 10722 log.cc:1196] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/48e85e8dba3c42e9bb1f4373e469d01b
I20260808 12:59:06.075680 10722 ts_tablet_manager.cc:1939] T 48e85e8dba3c42e9bb1f4373e469d01b P c92ff09f5c344d76bca2aa685faa7225: Deleting consensus metadata
I20260808 12:59:06.078161 9836 catalog_manager.cc:4841] TS c92ff09f5c344d76bca2aa685faa7225 (127.24.167.1:41381): tablet 48e85e8dba3c42e9bb1f4373e469d01b (table test-table [id=6159846038454ec6b392309acf925d6e]) successfully deleted
I20260808 12:59:06.081194 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:06.048820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.24.167.1:41381)
0808 12:59:06.049272 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:06.049278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:06.052343 (+ 3065us) server_negotiation.cc:407] Connection header received
0808 12:59:06.053440 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:06.053449 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:06.053577 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:06.053673 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:06.056433 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:06.058667 (+ 2234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:06.060904 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:06.061606 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:06.061724 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:06.069655 (+ 7931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:06.069671 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:06.069692 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:06.069822 (+ 130us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:06.076108 (+ 6286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:06.077822 (+ 1714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:06.077831 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:06.077834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:06.078195 (+ 361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:06.078909 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:06.078914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:06.078918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:06.079360 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:06.079817 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:06.080159 (+ 342us) server_negotiation.cc:299] Negotiation successful
0808 12:59:06.080504 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326}
I20260808 12:59:06.081826 10724 ts_tablet_manager.cc:1918] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 12:59:06.082132 10724 log.cc:1196] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/48e85e8dba3c42e9bb1f4373e469d01b
I20260808 12:59:06.082223 10174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:59:06.082995 10724 ts_tablet_manager.cc:1939] T 48e85e8dba3c42e9bb1f4373e469d01b P 043cff765f7d494a8a1386cc3451f2ba: Deleting consensus metadata
I20260808 12:59:06.085834 9831 catalog_manager.cc:4841] TS 043cff765f7d494a8a1386cc3451f2ba (127.24.167.3:45831): tablet 48e85e8dba3c42e9bb1f4373e469d01b (table test-table [id=6159846038454ec6b392309acf925d6e]) successfully deleted
I20260808 12:59:06.184719 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 9981
I20260808 12:59:06.212769 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 10191
Aug 08 12:59:06 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 12:59:06.229353 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:06.213820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.24.167.2:44809)
0808 12:59:06.214032 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:06.214036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:06.214054 (+ 18us) server_negotiation.cc:407] Connection header received
0808 12:59:06.214117 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:06.214120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:06.214183 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:06.214270 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:06.215726 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:06.216538 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:06.217878 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:06.218295 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:06.218545 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:06.223104 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:06.223123 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:06.223141 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:06.223180 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:06.226025 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:06.227009 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:06.227018 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:06.227020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:06.227155 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:06.227617 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:06.227621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:06.227623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:06.228056 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:06.228399 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:06.228721 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 12:59:06.228992 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 12:59:06.230121 10413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:59:06.322206 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 10191
I20260808 12:59:06.347918 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 10433
Aug 08 12:59:06 dist-test-slave-tk9z krb5kdc[9057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 12:59:06.364506 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:06.348830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53526 (local address 127.24.167.3:45831)
0808 12:59:06.349103 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:06.349108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:06.349132 (+ 24us) server_negotiation.cc:407] Connection header received
0808 12:59:06.349348 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:06.349353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:06.349451 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:06.349544 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:06.351024 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:06.351747 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:06.353038 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:06.353433 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:06.353535 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:06.357981 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:06.357999 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:06.358017 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:06.358054 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:06.361453 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:06.362252 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:06.362262 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:06.362264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:06.362497 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:06.362955 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:06.362958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:06.362960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:06.363358 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:06.363615 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:06.363918 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 12:59:06.364116 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260808 12:59:06.365164 10642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:59:06.456580 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 10433
I20260808 12:59:06.481462 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 9807
I20260808 12:59:06.482740 9912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:59:06.598073 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 9807
I20260808 12:59:06.633806 25244 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 12:59:06.979710 25244 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 12:59:06.980084 25244 mini_ranger.cc:66] Stopping Ranger...
I20260808 12:59:07.326766 25244 mini_ranger.cc:68] Stopped Ranger
2026-08-08 12:59:07.327 UTC [9102] LOG: received smart shutdown request
2026-08-08 12:59:07.331 UTC [9102] LOG: background worker "logical replication launcher" (PID 9118) exited with exit code 1
2026-08-08 12:59:07.331 UTC [9107] LOG: shutting down
2026-08-08 12:59:07.517 UTC [9102] LOG: database system is shut down
2026-08-08T12:59:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53963 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 12:59:07 dist-test-slave-tk9z krb5kdc[10744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 12:59:07 dist-test-slave-tk9z krb5kdc[10744](info): set up 2 sockets
Aug 08 12:59:07 dist-test-slave-tk9z krb5kdc[10744](info): commencing operation
krb5kdc: starting...
W20260808 12:59:09.006804 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.310s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 12:59:09 dist-test-slave-tk9z krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T12:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T12:59:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.167.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.167.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/rangeradmin_127.24.167.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/rangeradmin_127.24.167.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.167.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.167.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/rangerlookup_127.24.167.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/rangerlookup_127.24.167.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.167.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.167.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/HTTP_127.24.167.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/HTTP_127.24.167.20@KRBTEST.COM.keytab.
I20260808 12:59:09.400832 25244 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMF3Yup/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/postgres -l logfile start
2026-08-08 12:59:11.993 UTC [10799] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 12:59:11.994 UTC [10799] LOG: listening on IPv4 address "127.24.167.20", port 53949
2026-08-08 12:59:11.997 UTC [10799] LOG: listening on Unix socket "/tmp/.s.PGSQL.53949"
2026-08-08 12:59:12.020 UTC [10803] LOG: database system was shut down at 2026-08-08 12:59:10 UTC
2026-08-08 12:59:12.024 UTC [10799] LOG: database system is ready to accept connections
I20260808 12:59:13.216902 25244 mini_postgres.cc:96] Postgres bound to 53949
2026-08-08 12:59:13.239 UTC [10814] FATAL: database "slave" does not exist
127.24.167.20:53949 - accepting connections
I20260808 12:59:13.240059 25244 mini_ranger.cc:161] Starting Ranger...
I20260808 12:59:13.271123 25244 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 12:59:13.727684 25244 mini_ranger.cc:87] Created ranger Postgres database
I20260808 12:59:13.727794 25244 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-admin
2026-08-08 12:59:14,025 [I] DB FLAVOR :POSTGRES
2026-08-08 12:59:14,025 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 12:59:14,025 [I] Checking connection..
2026-08-08 12:59:14,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 12:59:14,291 [I] Checking connection passed.
2026-08-08 12:59:14,291 [I] --------- Verifying version history table ---------
2026-08-08 12:59:14,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 12:59:14,544 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 12:59:14,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 12:59:14,789 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 12:59:14,789 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 12:59:14,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 12:59:15.032 UTC [10915] WARNING: there is no transaction in progress
2026-08-08 12:59:15,041 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 12:59:15,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 12:59:15,312 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 12:59:15,312 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 12:59:15,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 12:59:15,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 12:59:15,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tk9z.c.gcp-upstream.internal','N') ;"
2026-08-08 12:59:16,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 12:59:16,331 [I] Table x_portal_user does not exist in database ranger
2026-08-08 12:59:16,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 12:59:16,598 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 12:59:16,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 12:59:16,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 12:59:17,108 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 12:59:17,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 12:59:18.049 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18.065 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18.081 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18.096 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18.111 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18.128 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18.181 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18.187 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18.200 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18.210 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18.220 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18.235 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18.244 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18.251 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18.258 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18.265 UTC [11072] WARNING: there is no transaction in progress
2026-08-08 12:59:18,878 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 12:59:18,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 12:59:19,113 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 12:59:19,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 12:59:19,353 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 12:59:19,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 12:59:19,611 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 12:59:19,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 12:59:19,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 12:59:20,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tk9z.c.gcp-upstream.internal';"
2026-08-08 12:59:20,389 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 12:59:20,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 12:59:20,649 [I] DB_PATCHES have already been applied
I20260808 12:59:20.655920 25244 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMF3Yup/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 12:59:20.656023 25244 mini_ranger.cc:193] Using host: 127.24.167.20
I20260808 12:59:20.676210 25244 mini_ranger.cc:227] Ranger admin URL: http://127.24.167.20:42607
Aug 08, 2026 12:59:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 12:59:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 12:59:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 12:59:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 12:59:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 12:59:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 12:59:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 12:59:22.084105 25244 mini_ranger.cc:160] Time spent starting Ranger: real 8.844s user 0.002s sys 0.054s
Aug 08, 2026 12:59:41 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 12:59:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 08, 2026 12:59:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 12:59:41 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 12:59:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 12:59:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 12:59:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 12:59:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 12:59:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 12:59:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 12:59:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 12:59:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 12:59:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 12:59:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 12:59:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 12:59:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 12:59:41 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 12:59:42 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 12:59:43.319212 25244 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.167.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.167.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/rangerkms_127.24.167.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.167.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/rangerkms_127.24.167.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.167.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.167.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.167.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/HTTP_127.24.167.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.167.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/HTTP_127.24.167.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 12:59:43 dist-test-slave-tk9z krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193983, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 12:59:43.520767 25244 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 12:59:43.546655 25244 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 12:59:44.331744 25244 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 12:59:44.331869 25244 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-kms
I20260808 12:59:44.331949 25244 mini_ranger_kms.cc:227] Using postgres at 127.24.167.20:53949
2026-08-08 12:59:44,690 [I] DB FLAVOR :POSTGRES
2026-08-08 12:59:44,690 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 12:59:44,690 [I] Checking connection
2026-08-08 12:59:44,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 12:59:44,958 [I] connection success
2026-08-08 12:59:44,959 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 12:59:44,959 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 12:59:44,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 12:59:45,226 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 12:59:45,226 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 12:59:45,226 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 12:59:45,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.167.20:53949/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 12:59:45,515 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 12:59:46.022405 25244 mini_ranger_kms.cc:310] Created kms service
I20260808 12:59:46.207818 25244 mini_ranger_kms.cc:326] Created kudu user
I20260808 12:59:46.255620 25244 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 12:59:46 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193983, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:59:46.679370 25244 mini_ranger_kms.cc:384] Added ranger policy
I20260808 12:59:46.679706 25244 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMF3Yup/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMF3Yup/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 12:59:46.679766 25244 mini_ranger_kms.cc:240] Using host: 127.24.167.20
I20260808 12:59:46.701825 25244 mini_ranger_kms.cc:276] Ranger KMS PID: 11361
I20260808 12:59:46.701968 25244 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.167.20:53161
Aug 08, 2026 12:59:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 12:59:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 12:59:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 12:59:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 12:59:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 12:59:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 12:59:49 dist-test-slave-tk9z krb5kdc[10744](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786193989, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.167.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:59:49.652107 25244 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.131s user 0.060s sys 0.331s
I20260808 12:59:49.652395 25244 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 12:59:49.652588 25244 mini_ranger_kms.cc:401] 127.24.167.20:53161/kms/v1/keys
Aug 08 12:59:49 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193989, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.167.20@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
Aug 08 12:59:50 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193989, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.167.20@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
Aug 08 12:59:52 dist-test-slave-tk9z krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 12:59:52.270545 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:35425
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:35425
--ranger_config_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.167.20:53161/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:59:52.611799 11441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:59:52.612131 11441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:59:52.612295 11441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:59:52.619675 11441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:59:52.619845 11441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:59:52.619943 11441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:59:52.620041 11441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:59:52.620131 11441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:59:52.630548 11441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.167.20:53161/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:35425
--ranger_config_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:35425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:59:52.631986 11441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:59:52.633395 11441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:59:52.645511 11451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:52.645571 11452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:52.645535 11455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:59:52.647778 11441 server_base.cc:1034] running on GCE node
I20260808 12:59:52.648619 11441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:59:52.650058 11441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:59:52.651298 11441 hybrid_clock.cc:648] HybridClock initialized: now 1786193992651260 us; error 47 us; skew 500 ppm
I20260808 12:59:52.651672 11441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:59:52 dist-test-slave-tk9z krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193992, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:59:52.654945 11441 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:59:52.656504 11441 webserver.cc:460] Webserver started at http://127.24.167.62:45919/ using document root <none> and password file <none>
I20260808 12:59:52.657195 11441 fs_manager.cc:359] Metadata directory not provided
I20260808 12:59:52.657331 11441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:59:52.657645 11441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 12:59:52 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193992, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:59:52.719300 11441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "3a2e12a24b5244b6b1b14775e508f061"
format_stamp: "Formatted at 2026-08-08 12:59:52 on dist-test-slave-tk9z"
server_key: "ccffa58f81f9b7fc18df3368fb394197"
server_key_iv: "af90fe487c794b8ec63259c0322a28f4"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:52.720469 11441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "3a2e12a24b5244b6b1b14775e508f061"
format_stamp: "Formatted at 2026-08-08 12:59:52 on dist-test-slave-tk9z"
server_key: "ccffa58f81f9b7fc18df3368fb394197"
server_key_iv: "af90fe487c794b8ec63259c0322a28f4"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:52.727720 11441 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260808 12:59:52.748638 11465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:52.750232 11441 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 12:59:52.750495 11441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "3a2e12a24b5244b6b1b14775e508f061"
format_stamp: "Formatted at 2026-08-08 12:59:52 on dist-test-slave-tk9z"
server_key: "ccffa58f81f9b7fc18df3368fb394197"
server_key_iv: "af90fe487c794b8ec63259c0322a28f4"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:52.750751 11441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:59:52.773406 11441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:59:52.778368 11441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:59:52.778834 11441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:59:52.799866 11441 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:35425
I20260808 12:59:52.799872 11542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:35425 every 8 connection(s)
I20260808 12:59:52.801510 11441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:59:52.805373 11543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:59:52.808779 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 11441
I20260808 12:59:52.809013 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 12:59:52.809689 25244 external_mini_cluster.cc:1428] Setting key e6d58fa5abd39dd632f51942d1136bbd
I20260808 12:59:52.816881 11543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061: Bootstrap starting.
I20260808 12:59:52.820696 11543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061: Neither blocks nor log segments found. Creating new log.
I20260808 12:59:52.821873 11543 log.cc:824] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061: Log is configured to *not* fsync() on all Append() calls
I20260808 12:59:52.825326 11543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061: No bootstrap required, opened a new log
Aug 08 12:59:52 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:59:52.834088 11548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:52.811897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44088 (local address 127.24.167.62:35425)
0808 12:59:52.812750 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:52.812762 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:52.812806 (+ 44us) server_negotiation.cc:407] Connection header received
0808 12:59:52.815223 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:52.815250 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:52.815672 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:52.815993 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:52.817748 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:52.819135 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:52.821055 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:52.821749 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:52.821893 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:52.826703 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:52.826717 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:52.826732 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:52.826773 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:52.829906 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:52.830792 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:52.830802 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:52.830808 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:52.830958 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:52.831361 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:52.831364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:52.831366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:52.831932 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:52.832186 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:52.832647 (+ 461us) server_negotiation.cc:299] Negotiation successful
0808 12:59:52.833042 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":408,"threads_started":1}
I20260808 12:59:52.834340 11543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2e12a24b5244b6b1b14775e508f061" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35425 } }
I20260808 12:59:52.834868 11543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:59:52.835000 11543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2e12a24b5244b6b1b14775e508f061, State: Initialized, Role: FOLLOWER
I20260808 12:59:52.835637 11543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2e12a24b5244b6b1b14775e508f061" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35425 } }
I20260808 12:59:52.835911 11543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:59:52.836078 11543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:59:52.836251 11543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:59:52.838791 11543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2e12a24b5244b6b1b14775e508f061" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35425 } }
I20260808 12:59:52.839442 11543 leader_election.cc:302] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a2e12a24b5244b6b1b14775e508f061; no voters:
I20260808 12:59:52.840072 11543 leader_election.cc:288] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 12:59:52.840195 11551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:59:52.840956 11551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 1 LEADER]: Becoming Leader. State: Replica: 3a2e12a24b5244b6b1b14775e508f061, State: Running, Role: LEADER
I20260808 12:59:52.841460 11551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2e12a24b5244b6b1b14775e508f061" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35425 } }
I20260808 12:59:52.842198 11543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:59:52.844417 11553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a2e12a24b5244b6b1b14775e508f061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2e12a24b5244b6b1b14775e508f061" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35425 } } }
I20260808 12:59:52.844411 11554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a2e12a24b5244b6b1b14775e508f061. Latest consensus state: current_term: 1 leader_uuid: "3a2e12a24b5244b6b1b14775e508f061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2e12a24b5244b6b1b14775e508f061" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35425 } } }
I20260808 12:59:52.844892 11553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [sys.catalog]: This master's current role is: LEADER
I20260808 12:59:52.844892 11554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [sys.catalog]: This master's current role is: LEADER
I20260808 12:59:52.845789 11559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:59:52.851783 11559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:59:52.863391 11559 catalog_manager.cc:1347] Generated new cluster ID: 14929179b8f048e29a44892e9c2122a7
I20260808 12:59:52.863548 11559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:59:52.867821 11543 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 12:59:52.893194 11559 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 12:59:52.896184 11559 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:59:52.906386 11559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061: Generated new TSK 0
I20260808 12:59:52.907622 11559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 12:59:53.662537 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.167.20:53161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35425
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 12:59:53 dist-test-slave-tk9z krb5kdc[10744](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786193993, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 12:59:54.092741 11618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:59:54.093124 11618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:59:54.093308 11618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:59:54.101179 11618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:59:54.101353 11618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:59:54.101543 11618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:59:54.112705 11618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.167.20:53161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:59:54.114135 11618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:59:54.115695 11618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:59:54.135921 11630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:54.139498 11631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:54.148825 11634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:59:54.152840 11618 server_base.cc:1034] running on GCE node
I20260808 12:59:54.153888 11618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:59:54.157630 11618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:59:54.158972 11618 hybrid_clock.cc:648] HybridClock initialized: now 1786193994158850 us; error 117 us; skew 500 ppm
I20260808 12:59:54.159458 11618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:59:54 dist-test-slave-tk9z krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193994, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:59:54.164135 11618 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 12:59:54.172066 11618 webserver.cc:460] Webserver started at http://127.24.167.1:39505/ using document root <none> and password file <none>
I20260808 12:59:54.173092 11618 fs_manager.cc:359] Metadata directory not provided
I20260808 12:59:54.173249 11618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:59:54.173640 11618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 12:59:54 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193994, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:59:54.195653 11618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "c010643f84f648dc8660dca25fde1fa9"
format_stamp: "Formatted at 2026-08-08 12:59:54 on dist-test-slave-tk9z"
server_key: "2d3da46d967e25dc90e23f0334be01cb"
server_key_iv: "eeac732c91e27343e426bfd66085ea3e"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:54.196957 11618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "c010643f84f648dc8660dca25fde1fa9"
format_stamp: "Formatted at 2026-08-08 12:59:54 on dist-test-slave-tk9z"
server_key: "2d3da46d967e25dc90e23f0334be01cb"
server_key_iv: "eeac732c91e27343e426bfd66085ea3e"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:54.231542 11618 fs_manager.cc:691] Time spent creating directory manager: real 0.034s user 0.016s sys 0.016s
I20260808 12:59:54.252633 11643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:54.254734 11618 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 12:59:54.254942 11618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "c010643f84f648dc8660dca25fde1fa9"
format_stamp: "Formatted at 2026-08-08 12:59:54 on dist-test-slave-tk9z"
server_key: "2d3da46d967e25dc90e23f0334be01cb"
server_key_iv: "eeac732c91e27343e426bfd66085ea3e"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:54.255223 11618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:59:54.294440 11618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:59:54.300076 11618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:59:54.300460 11618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:59:54.301792 11618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:59:54.303727 11618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:59:54.303829 11618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:54.304010 11618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:59:54.304080 11618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:54.402029 11618 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:37687
I20260808 12:59:54.402068 11821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:37687 every 8 connection(s)
I20260808 12:59:54.403973 11618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:59:54.410230 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 11618
I20260808 12:59:54.410531 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 12:59:54.411649 25244 external_mini_cluster.cc:1428] Setting key 07178e47bc540ff6bac815291e942be1
Aug 08 12:59:54 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193994, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
I20260808 12:59:54.475016 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:54.410827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:35425 (local address 127.24.167.1:47519)
0808 12:59:54.436605 (+ 25778us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:54.436675 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:54.443591 (+ 6916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:54.444820 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:54.444834 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:54.445584 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:59:54.446522 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:54.446549 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:54.450473 (+ 3924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:54.450479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:54.452786 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:54.452805 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:54.454939 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:54.454946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:54.455327 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:54.456207 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:59:54.456239 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:59:54.463931 (+ 7692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:59:54.469053 (+ 5122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:54.469065 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:54.469085 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:54.469790 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:54.470305 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:54.470310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:54.470613 (+ 303us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:54.470912 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:54.471833 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:59:54.471845 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:59:54.472392 (+ 547us) client_negotiation.cc:769] Sending connection context
0808 12:59:54.472793 (+ 401us) client_negotiation.cc:240] Negotiation successful
0808 12:59:54.473415 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25333,"thread_start_us":27419,"threads_started":1}
I20260808 12:59:54.475368 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:54.410179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:47519 (local address 127.24.167.62:35425)
0808 12:59:54.443274 (+ 33095us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:54.443281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:54.443313 (+ 32us) server_negotiation.cc:407] Connection header received
0808 12:59:54.444096 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:54.444102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:54.444217 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:54.444321 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:54.446745 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:54.447943 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:54.453677 (+ 5734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:54.454456 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:54.454561 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:54.464301 (+ 9740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:54.464309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:54.464314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:54.464376 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:54.468197 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:54.469976 (+ 1779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:54.469984 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:54.469988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:54.470158 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:54.471073 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:54.471081 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:54.471087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:54.471585 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:54.471746 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:54.474646 (+ 2900us) server_negotiation.cc:299] Negotiation successful
0808 12:59:54.474911 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32944,"thread_start_us":32799,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
I20260808 12:59:54.479017 11823 heartbeater.cc:347] Connected to a master server at 127.24.167.62:35425
I20260808 12:59:54.479382 11823 heartbeater.cc:464] Registering TS with master...
I20260808 12:59:54.480273 11823 heartbeater.cc:511] Master 127.24.167.62:35425 requested a full tablet report, sending...
I20260808 12:59:54.483103 11490 ts_manager.cc:194] Registered new tserver with Master: c010643f84f648dc8660dca25fde1fa9 (127.24.167.1:37687)
I20260808 12:59:54.485213 11490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:47519
Aug 08 12:59:54 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193993, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
W20260808 12:59:54.517302 25244 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.24.167.2: real 0.101s user 0.001s sys 0.022s
I20260808 12:59:54.518293 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.167.20:53161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35425
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 12:59:54 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193993, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
W20260808 12:59:55.055090 11836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:59:55.055594 11836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:59:55.055807 11836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:59:55.067619 11836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:59:55.067883 11836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:59:55.068151 11836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:59:55.083891 11836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.167.20:53161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:59:55.085907 11836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:59:55.087810 11836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:59:55.123597 11847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:55.128136 11848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:55.142204 11854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:59:55.144134 11836 server_base.cc:1034] running on GCE node
I20260808 12:59:55.145205 11836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:59:55.146662 11836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:59:55.147931 11836 hybrid_clock.cc:648] HybridClock initialized: now 1786193995147820 us; error 98 us; skew 500 ppm
I20260808 12:59:55.148391 11836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:59:55 dist-test-slave-tk9z krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193995, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:59:55.153194 11836 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 12:59:55.193090 11836 webserver.cc:460] Webserver started at http://127.24.167.2:35013/ using document root <none> and password file <none>
I20260808 12:59:55.194410 11836 fs_manager.cc:359] Metadata directory not provided
I20260808 12:59:55.194720 11836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:59:55.195199 11836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 12:59:55 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193995, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:59:55.222417 11836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "eddfdf820ac84ab68137ba8fcd57f130"
format_stamp: "Formatted at 2026-08-08 12:59:55 on dist-test-slave-tk9z"
server_key: "f6bba3ff29a8f3ead62a9b32419ddcaa"
server_key_iv: "6c6d786554af701956fec3f3eae82cab"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:55.225183 11836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "eddfdf820ac84ab68137ba8fcd57f130"
format_stamp: "Formatted at 2026-08-08 12:59:55 on dist-test-slave-tk9z"
server_key: "f6bba3ff29a8f3ead62a9b32419ddcaa"
server_key_iv: "6c6d786554af701956fec3f3eae82cab"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:55.239647 11836 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.008s sys 0.004s
I20260808 12:59:55.282159 11863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:55.284395 11836 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.006s sys 0.009s
I20260808 12:59:55.284713 11836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "eddfdf820ac84ab68137ba8fcd57f130"
format_stamp: "Formatted at 2026-08-08 12:59:55 on dist-test-slave-tk9z"
server_key: "f6bba3ff29a8f3ead62a9b32419ddcaa"
server_key_iv: "6c6d786554af701956fec3f3eae82cab"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:55.285141 11836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:59:55.317214 11836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:59:55.321380 11836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:59:55.321697 11836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:59:55.322839 11836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:59:55.324409 11836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:59:55.324487 11836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:55.324661 11836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:59:55.324716 11836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:55.366364 11836 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:40057
I20260808 12:59:55.366422 12041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:40057 every 8 connection(s)
I20260808 12:59:55.367745 11836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:59:55.368268 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 11836
I20260808 12:59:55.368505 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 12:59:55.369328 25244 external_mini_cluster.cc:1428] Setting key dc9189d50382d9c0fc00b1186bb7f680
Aug 08 12:59:55 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193995, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
I20260808 12:59:55.400563 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:55.371187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:54241 (local address 127.24.167.62:35425)
0808 12:59:55.372887 (+ 1700us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:55.372893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:55.373604 (+ 711us) server_negotiation.cc:407] Connection header received
0808 12:59:55.376282 (+ 2678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:55.376291 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:55.376401 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:55.376526 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:55.378427 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:55.379647 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:55.381943 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:55.382591 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:55.382741 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:55.392328 (+ 9587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:55.392336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:55.392341 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:55.392390 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:55.396454 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:55.397548 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:55.397556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:55.397560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:55.397747 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:55.398326 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:55.398334 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:55.398340 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:55.398812 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:55.398990 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:55.399933 (+ 943us) server_negotiation.cc:299] Negotiation successful
0808 12:59:55.400224 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1566,"thread_start_us":1444,"threads_started":1}
I20260808 12:59:55.402324 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:55.372017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:35425 (local address 127.24.167.2:54241)
0808 12:59:55.373416 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:55.373477 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:55.375958 (+ 2481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:55.376785 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:55.376807 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:55.377404 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:59:55.378187 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:55.378210 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:55.379830 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:55.379835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:55.381770 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:55.381785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:55.382844 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:55.382853 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:55.383109 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:55.388386 (+ 5277us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:59:55.388431 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:59:55.391953 (+ 3522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:59:55.396699 (+ 4746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:55.396711 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:55.396732 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:55.397391 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:55.397893 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:55.397897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:55.397901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:55.398169 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:55.398986 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:59:55.398995 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:59:55.399467 (+ 472us) client_negotiation.cc:769] Sending connection context
0808 12:59:55.399787 (+ 320us) client_negotiation.cc:240] Negotiation successful
0808 12:59:55.400417 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1106,"thread_start_us":2245,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
I20260808 12:59:55.404474 12042 heartbeater.cc:347] Connected to a master server at 127.24.167.62:35425
I20260808 12:59:55.404819 12042 heartbeater.cc:464] Registering TS with master...
I20260808 12:59:55.405673 12042 heartbeater.cc:511] Master 127.24.167.62:35425 requested a full tablet report, sending...
I20260808 12:59:55.407460 11490 ts_manager.cc:194] Registered new tserver with Master: eddfdf820ac84ab68137ba8fcd57f130 (127.24.167.2:40057)
I20260808 12:59:55.408545 11490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:54241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 12:59:55.437426 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.167.20:53161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35425
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 12:59:55.489480 11823 heartbeater.cc:503] Master 127.24.167.62:35425 was elected leader, sending a full tablet report...
W20260808 12:59:55.843989 12055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:59:55.844441 12055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:59:55.844621 12055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:59:55.853950 12055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:59:55.854144 12055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:59:55.854447 12055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:59:55.866919 12055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.167.20:53161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:35425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:59:55.868360 12055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:59:55.869801 12055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:59:55.885892 12065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:55.886075 12071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:55.886734 12066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:59:55.888149 12055 server_base.cc:1034] running on GCE node
I20260808 12:59:55.888864 12055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:59:55.890066 12055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:59:55.891458 12055 hybrid_clock.cc:648] HybridClock initialized: now 1786193995891357 us; error 51 us; skew 500 ppm
I20260808 12:59:55.891986 12055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:59:55 dist-test-slave-tk9z krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193995, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:59:55.897059 12055 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 12:59:55.898957 12055 webserver.cc:460] Webserver started at http://127.24.167.3:39009/ using document root <none> and password file <none>
I20260808 12:59:55.899942 12055 fs_manager.cc:359] Metadata directory not provided
I20260808 12:59:55.900156 12055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 12:59:55.900557 12055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 12:59:55 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193995, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:59:55.917269 12055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "6a4b66fc87824f7f84d3f537aadfdec8"
format_stamp: "Formatted at 2026-08-08 12:59:55 on dist-test-slave-tk9z"
server_key: "2bb8545b942276db2dbdbcf28ab4b05e"
server_key_iv: "cf4fedbf7cf77646987eabf7de56875a"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:55.918300 12055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "6a4b66fc87824f7f84d3f537aadfdec8"
format_stamp: "Formatted at 2026-08-08 12:59:55 on dist-test-slave-tk9z"
server_key: "2bb8545b942276db2dbdbcf28ab4b05e"
server_key_iv: "cf4fedbf7cf77646987eabf7de56875a"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:55.925022 12055 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 12:59:55.941418 12079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:55.942981 12055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:59:55.943215 12055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "6a4b66fc87824f7f84d3f537aadfdec8"
format_stamp: "Formatted at 2026-08-08 12:59:55 on dist-test-slave-tk9z"
server_key: "2bb8545b942276db2dbdbcf28ab4b05e"
server_key_iv: "cf4fedbf7cf77646987eabf7de56875a"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:55.943487 12055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:59:55.977191 12055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:59:55.982093 12055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:59:55.982539 12055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:59:55.984050 12055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:59:55.985961 12055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:59:55.986100 12055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:55.986343 12055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:59:55.986521 12055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260808 12:59:56.057381 12055 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:41709
I20260808 12:59:56.057416 12258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:41709 every 8 connection(s)
I20260808 12:59:56.059134 12055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:59:56.067957 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 12055
I20260808 12:59:56.068154 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 12:59:56.068588 25244 external_mini_cluster.cc:1428] Setting key 01927e71be085cf1079796d8a09e9a74
Aug 08 12:59:56 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193995, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:59:56.083811 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:56.062295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:60585 (local address 127.24.167.62:35425)
0808 12:59:56.063021 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:56.063026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:56.064552 (+ 1526us) server_negotiation.cc:407] Connection header received
0808 12:59:56.066977 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:56.066983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:56.067113 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:56.067234 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:56.069097 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:56.070054 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:56.071949 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:56.072623 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:56.072759 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:56.076727 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:56.076732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:56.076736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:56.076767 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:56.080395 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:56.081200 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:56.081205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:56.081207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:56.081358 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:56.081879 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:56.081885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:56.081889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:56.082153 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:56.082350 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:56.083015 (+ 665us) server_negotiation.cc:299] Negotiation successful
0808 12:59:56.083304 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":421,"threads_started":1}
I20260808 12:59:56.084820 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:56.063266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:35425 (local address 127.24.167.3:60585)
0808 12:59:56.064375 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:56.064415 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:56.066704 (+ 2289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:56.067508 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:56.067517 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:56.068031 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:59:56.068921 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:56.068946 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:56.070273 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:56.070278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:56.071800 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:56.071812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:56.072901 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:56.072910 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:56.073217 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:56.073889 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:59:56.073911 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:59:56.076496 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:59:56.080536 (+ 4040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:56.080545 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:56.080560 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:56.081036 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:56.081478 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:56.081484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:56.081488 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:56.081746 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:56.082297 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:59:56.082305 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:59:56.082645 (+ 340us) client_negotiation.cc:769] Sending connection context
0808 12:59:56.082901 (+ 256us) client_negotiation.cc:240] Negotiation successful
0808 12:59:56.083486 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":521,"threads_started":1}
I20260808 12:59:56.086194 12259 heartbeater.cc:347] Connected to a master server at 127.24.167.62:35425
I20260808 12:59:56.086653 12259 heartbeater.cc:464] Registering TS with master...
I20260808 12:59:56.087543 12259 heartbeater.cc:511] Master 127.24.167.62:35425 requested a full tablet report, sending...
I20260808 12:59:56.089174 11490 ts_manager.cc:194] Registered new tserver with Master: 6a4b66fc87824f7f84d3f537aadfdec8 (127.24.167.3:41709)
I20260808 12:59:56.090226 11490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:60585
I20260808 12:59:56.097323 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 12:59:56.099232 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 11618
Aug 08 12:59:56 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 12:59:56.119385 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:56.100191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.24.167.1:37687)
0808 12:59:56.101569 (+ 1378us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:56.101576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:56.101602 (+ 26us) server_negotiation.cc:407] Connection header received
0808 12:59:56.101662 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:56.101669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:56.101879 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:56.102011 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:56.103873 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:56.105153 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:56.107148 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:56.107822 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:56.107928 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:56.113059 (+ 5131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:56.113072 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:56.113085 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:56.113133 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:56.116540 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:56.117306 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:56.117313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:56.117315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:56.117526 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:56.117899 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:56.117903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:56.117905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:56.118293 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:56.118643 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:56.118784 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 12:59:56.118983 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1207,"thread_start_us":999,"threads_started":1}
I20260808 12:59:56.120756 11817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:59:56.206982 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 11618
I20260808 12:59:56.231323 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 11836
Aug 08 12:59:56 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 12:59:56.253890 12271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:56.236666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42524 (local address 127.24.167.2:40057)
0808 12:59:56.237233 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:56.237241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:56.237267 (+ 26us) server_negotiation.cc:407] Connection header received
0808 12:59:56.237347 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:56.237354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:56.237585 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:56.237720 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:56.239152 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:56.240305 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:56.241682 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:56.242517 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:56.242723 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:56.247824 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:56.247843 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:56.247863 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:56.247937 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:56.251043 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:56.251838 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:56.251846 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:56.251848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:56.251995 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:56.252298 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:56.252301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:56.252303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:56.252753 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:56.252986 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:56.253260 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 12:59:56.253493 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":317,"threads_started":1}
I20260808 12:59:56.255295 12037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:59:56.337744 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 11836
I20260808 12:59:56.366632 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 12055
Aug 08 12:59:56 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 12:59:56.383798 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:56.367731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54260 (local address 127.24.167.3:41709)
0808 12:59:56.368361 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:56.368369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:56.368401 (+ 32us) server_negotiation.cc:407] Connection header received
0808 12:59:56.368467 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:56.368473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:56.368723 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:56.368862 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:56.370225 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:56.371620 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:56.372912 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:56.373521 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:56.373629 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:56.378127 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:56.378139 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:56.378155 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:56.378197 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:56.381184 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:56.381811 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:56.381817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:56.381820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:56.381961 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:56.382270 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:56.382273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:56.382275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:56.382681 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:56.382927 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:56.383148 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 12:59:56.383356 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":277,"threads_started":1}
I20260808 12:59:56.385063 12254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:59:56.468559 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 12055
I20260808 12:59:56.493558 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 11441
I20260808 12:59:56.494874 11536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 12:59:56.604180 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 11441
I20260808 12:59:56.636909 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:35425
--webserver_interface=127.24.167.62
--webserver_port=45919
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:35425
--ranger_config_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.167.20:53161/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.167.20:53161/kms with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 12:59:56.987695 12284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:59:56.988015 12284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:59:56.988199 12284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:59:56.995610 12284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 12:59:56.995738 12284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:59:56.995821 12284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:59:56.995916 12284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 12:59:56.995991 12284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 12:59:57.006064 12284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.167.20:53161/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:35425
--ranger_config_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:35425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=45919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:59:57.007483 12284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:59:57.008846 12284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:59:57.020993 12292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:57.020983 12296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:57.021044 12293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:59:57.021839 12284 server_base.cc:1034] running on GCE node
I20260808 12:59:57.022691 12284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:59:57.024156 12284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:59:57.025388 12284 hybrid_clock.cc:648] HybridClock initialized: now 1786193997025340 us; error 42 us; skew 500 ppm
I20260808 12:59:57.025768 12284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:59:57 dist-test-slave-tk9z krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193997, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:59:57.028962 12284 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 12:59:57.030659 12284 webserver.cc:460] Webserver started at http://127.24.167.62:45919/ using document root <none> and password file <none>
I20260808 12:59:57.031332 12284 fs_manager.cc:359] Metadata directory not provided
I20260808 12:59:57.031487 12284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 12:59:57 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193997, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:59:57.112533 12284 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 12:59:57.115731 12309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:57.117197 12284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 12:59:57.117401 12284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "3a2e12a24b5244b6b1b14775e508f061"
format_stamp: "Formatted at 2026-08-08 12:59:52 on dist-test-slave-tk9z"
server_key: "ccffa58f81f9b7fc18df3368fb394197"
server_key_iv: "af90fe487c794b8ec63259c0322a28f4"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:57.118467 12284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:59:57.144373 12284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:59:57.148348 12284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:59:57.148689 12284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:59:57.167616 12284 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:35425
I20260808 12:59:57.167641 12396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:35425 every 8 connection(s)
I20260808 12:59:57.169206 12284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 12:59:57.174661 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 12284
I20260808 12:59:57.175822 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.167.20:53161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:37687
--local_ip_for_outbound_sockets=127.24.167.1
--tserver_master_addrs=127.24.167.62:35425
--webserver_port=39505
--webserver_interface=127.24.167.1
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.167.20:53161/kms with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 12:59:57.177120 12397 sys_catalog.cc:263] Verifying existing consensus state
I20260808 12:59:57.182395 12397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061: Bootstrap starting.
I20260808 12:59:57.201960 12397 log.cc:824] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061: Log is configured to *not* fsync() on all Append() calls
I20260808 12:59:57.211102 12397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 12:59:57.211655 12397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061: Bootstrap complete.
I20260808 12:59:57.221069 12397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2e12a24b5244b6b1b14775e508f061" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35425 } }
I20260808 12:59:57.221969 12397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2e12a24b5244b6b1b14775e508f061, State: Initialized, Role: FOLLOWER
I20260808 12:59:57.222676 12397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2e12a24b5244b6b1b14775e508f061" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35425 } }
I20260808 12:59:57.222924 12397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 12:59:57.223071 12397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 12:59:57.223268 12397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 1 FOLLOWER]: Advancing to term 2
I20260808 12:59:57.227226 12397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2e12a24b5244b6b1b14775e508f061" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35425 } }
I20260808 12:59:57.227756 12397 leader_election.cc:302] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a2e12a24b5244b6b1b14775e508f061; no voters:
I20260808 12:59:57.228387 12397 leader_election.cc:288] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 12:59:57.228531 12402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 12:59:57.229880 12402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [term 2 LEADER]: Becoming Leader. State: Replica: 3a2e12a24b5244b6b1b14775e508f061, State: Running, Role: LEADER
I20260808 12:59:57.230479 12402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2e12a24b5244b6b1b14775e508f061" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35425 } }
I20260808 12:59:57.230728 12397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 12:59:57.233175 12403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3a2e12a24b5244b6b1b14775e508f061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2e12a24b5244b6b1b14775e508f061" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35425 } } }
I20260808 12:59:57.233636 12403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [sys.catalog]: This master's current role is: LEADER
I20260808 12:59:57.237536 12406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a2e12a24b5244b6b1b14775e508f061. Latest consensus state: current_term: 2 leader_uuid: "3a2e12a24b5244b6b1b14775e508f061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2e12a24b5244b6b1b14775e508f061" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35425 } } }
I20260808 12:59:57.237892 12406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061 [sys.catalog]: This master's current role is: LEADER
I20260808 12:59:57.239653 12411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 12:59:57.245409 12411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 12:59:57.246886 12411 catalog_manager.cc:1259] Loaded cluster ID: 14929179b8f048e29a44892e9c2122a7
I20260808 12:59:57.246976 12411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 12:59:57.251027 12411 catalog_manager.cc:1504] Loading token signing keys...
I20260808 12:59:57.252434 12411 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061: Loaded TSK: 0
I20260808 12:59:57.253937 12411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 12:59:57.257771 12397 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260808 12:59:57.633067 12399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:59:57.633415 12399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:59:57.633567 12399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:59:57.644838 12399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:59:57.645015 12399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:59:57.645187 12399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 12:59:57.661301 12399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.167.20:53161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:37687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=39505
--tserver_master_addrs=127.24.167.62:35425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:59:57.663278 12399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:59:57.665062 12399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:59:57.687860 12455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:57.691469 12456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:59:57.698665 12399 server_base.cc:1034] running on GCE node
W20260808 12:59:57.699723 12463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:59:57.700739 12399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:59:57.702081 12399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:59:57.703305 12399 hybrid_clock.cc:648] HybridClock initialized: now 1786193997703208 us; error 84 us; skew 500 ppm
I20260808 12:59:57.704041 12399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:59:57 dist-test-slave-tk9z krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193997, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:59:57.709182 12399 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 12:59:57.724654 12399 webserver.cc:460] Webserver started at http://127.24.167.1:39505/ using document root <none> and password file <none>
I20260808 12:59:57.725744 12399 fs_manager.cc:359] Metadata directory not provided
I20260808 12:59:57.725943 12399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 12:59:57 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193997, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:59:57.761386 12399 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.004s sys 0.004s
I20260808 12:59:57.771308 12476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:57.773418 12399 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.005s
I20260808 12:59:57.773655 12399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "c010643f84f648dc8660dca25fde1fa9"
format_stamp: "Formatted at 2026-08-08 12:59:54 on dist-test-slave-tk9z"
server_key: "2d3da46d967e25dc90e23f0334be01cb"
server_key_iv: "eeac732c91e27343e426bfd66085ea3e"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:57.775285 12399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:59:57.837126 12399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:59:57.842489 12399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:59:57.842808 12399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:59:57.847182 12399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:59:57.848891 12399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:59:57.849059 12399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:57.849193 12399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:59:57.849283 12399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:57.933110 12399 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:37687
I20260808 12:59:57.933182 12663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:37687 every 8 connection(s)
I20260808 12:59:57.934901 12399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 12:59:57.942157 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 12399
I20260808 12:59:57.943122 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.167.20:53161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:40057
--local_ip_for_outbound_sockets=127.24.167.2
--tserver_master_addrs=127.24.167.62:35425
--webserver_port=35013
--webserver_interface=127.24.167.2
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.167.20:53161/kms with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 12:59:57 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193997, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:59:57.973212 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:57.938836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:52575 (local address 127.24.167.62:35425)
0808 12:59:57.942791 (+ 3955us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:57.942812 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:57.942861 (+ 49us) server_negotiation.cc:407] Connection header received
0808 12:59:57.943382 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:57.943407 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:57.943884 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:57.944311 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:57.946304 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:57.948753 (+ 2449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:57.951330 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:57.952502 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:57.952719 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:57.957307 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:57.957319 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:57.957342 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:57.957398 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:57.967658 (+ 10260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:57.968625 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:57.968633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:57.968639 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:57.968809 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:57.969344 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:57.969349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:57.969352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:57.969920 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:57.970143 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:57.971237 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0808 12:59:57.971631 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3670,"thread_start_us":3399,"threads_started":1}
I20260808 12:59:57.973971 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:57.938809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:35425 (local address 127.24.167.1:52575)
0808 12:59:57.939718 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:57.939768 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:57.941835 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:57.944511 (+ 2676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:57.944522 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:57.945174 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:59:57.946025 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:57.946053 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:57.949001 (+ 2948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:57.949007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:57.951153 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:57.951172 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:57.952777 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:57.952784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:57.953088 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:57.953997 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:59:57.954029 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:59:57.956931 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:59:57.967866 (+ 10935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:57.967878 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:57.967898 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:57.968415 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:57.968959 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:57.968963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:57.968965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:57.969190 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:57.970079 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:59:57.970089 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:59:57.970598 (+ 509us) client_negotiation.cc:769] Sending connection context
0808 12:59:57.970988 (+ 390us) client_negotiation.cc:240] Negotiation successful
0808 12:59:57.971617 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":387,"threads_started":1}
I20260808 12:59:57.977800 12665 heartbeater.cc:347] Connected to a master server at 127.24.167.62:35425
I20260808 12:59:57.978185 12665 heartbeater.cc:464] Registering TS with master...
I20260808 12:59:57.979297 12665 heartbeater.cc:511] Master 127.24.167.62:35425 requested a full tablet report, sending...
W20260808 12:59:57.982463 12665 heartbeater.cc:644] Failed to heartbeat to 127.24.167.62:35425 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 12:59:57.982699 12665 heartbeater.cc:464] Registering TS with master...
I20260808 12:59:57.982991 12665 heartbeater.cc:511] Master 127.24.167.62:35425 requested a full tablet report, sending...
I20260808 12:59:57.984123 12665 heartbeater.cc:464] Registering TS with master...
I20260808 12:59:57.984388 12665 heartbeater.cc:511] Master 127.24.167.62:35425 requested a full tablet report, sending...
W20260808 12:59:57.985456 12665 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 12:59:58.427459 12671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:59:58.427763 12671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:59:58.427943 12671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:59:58.437201 12671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:59:58.437387 12671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:59:58.437597 12671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 12:59:58.450562 12671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.167.20:53161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:40057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=35013
--tserver_master_addrs=127.24.167.62:35425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:59:58.452036 12671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:59:58.453420 12671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:59:58.469954 12695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:58.470057 12698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:58.470624 12694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:59:58.472313 12671 server_base.cc:1034] running on GCE node
I20260808 12:59:58.473166 12671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:59:58.474951 12671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:59:58.476219 12671 hybrid_clock.cc:648] HybridClock initialized: now 1786193998476158 us; error 60 us; skew 500 ppm
I20260808 12:59:58.476640 12671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:59:58 dist-test-slave-tk9z krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193998, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:59:58.480693 12671 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 12:59:58.482753 12671 webserver.cc:460] Webserver started at http://127.24.167.2:35013/ using document root <none> and password file <none>
I20260808 12:59:58.483613 12671 fs_manager.cc:359] Metadata directory not provided
I20260808 12:59:58.483791 12671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 12:59:58 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193998, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:59:58.529769 12671 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260808 12:59:58.533816 12714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:58.535792 12671 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260808 12:59:58.536072 12671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "eddfdf820ac84ab68137ba8fcd57f130"
format_stamp: "Formatted at 2026-08-08 12:59:55 on dist-test-slave-tk9z"
server_key: "f6bba3ff29a8f3ead62a9b32419ddcaa"
server_key_iv: "6c6d786554af701956fec3f3eae82cab"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:58.537626 12671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:59:58.580639 12671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:59:58.585062 12671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:59:58.585384 12671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:59:58.586578 12671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:59:58.588217 12671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:59:58.588340 12671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:58.588474 12671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:59:58.588572 12671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 12:59:58 dist-test-slave-tk9z krb5kdc[10744](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786193998, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:59:58.673295 12671 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:40057
I20260808 12:59:58.673349 12895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:40057 every 8 connection(s)
I20260808 12:59:58.674988 12671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 12:59:58.681332 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 12671
I20260808 12:59:58.682032 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.167.20:53161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:41709
--local_ip_for_outbound_sockets=127.24.167.3
--tserver_master_addrs=127.24.167.62:35425
--webserver_port=39009
--webserver_interface=127.24.167.3
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.167.20:53161/kms with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 12:59:58 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193998, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:59:58.705150 12898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:58.678823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:50577 (local address 127.24.167.62:35425)
0808 12:59:58.679941 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:58.679960 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:58.683500 (+ 3540us) server_negotiation.cc:407] Connection header received
0808 12:59:58.686600 (+ 3100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:58.686609 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:58.686716 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:58.686806 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:58.689342 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:58.690632 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:58.693077 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:58.693776 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:58.693927 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:58.697954 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:58.697960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:58.697963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:58.697999 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:58.701773 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:58.702614 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:58.702619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:58.702627 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:58.702757 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:58.703262 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:58.703266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:58.703268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:58.703559 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:58.703721 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:58.704596 (+ 875us) server_negotiation.cc:299] Negotiation successful
0808 12:59:58.704848 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981,"thread_start_us":1831,"threads_started":1}
I20260808 12:59:58.707147 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:58.680478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:35425 (local address 127.24.167.2:50577)
0808 12:59:58.683323 (+ 2845us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:58.683374 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:58.686213 (+ 2839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:58.687448 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:58.687463 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:58.688198 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:59:58.689109 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:58.689141 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:58.690837 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:58.690843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:58.692876 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:58.692893 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:58.694069 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:58.694077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:58.694403 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:58.695260 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:59:58.695293 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:59:58.697702 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:59:58.701923 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:58.701933 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:58.701953 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:58.702484 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:58.702876 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:58.702881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:58.702885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:58.703128 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:58.703696 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:59:58.703704 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:59:58.704122 (+ 418us) client_negotiation.cc:769] Sending connection context
0808 12:59:58.705490 (+ 1368us) client_negotiation.cc:240] Negotiation successful
0808 12:59:58.705955 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2508,"thread_start_us":2238,"threads_started":1}
I20260808 12:59:58.708880 12896 heartbeater.cc:347] Connected to a master server at 127.24.167.62:35425
I20260808 12:59:58.709219 12896 heartbeater.cc:464] Registering TS with master...
I20260808 12:59:58.709982 12896 heartbeater.cc:511] Master 127.24.167.62:35425 requested a full tablet report, sending...
I20260808 12:59:58.711884 12343 ts_manager.cc:194] Registered new tserver with Master: eddfdf820ac84ab68137ba8fcd57f130 (127.24.167.2:40057)
I20260808 12:59:58.713478 12343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:50577
I20260808 12:59:58.985618 12665 heartbeater.cc:464] Registering TS with master...
I20260808 12:59:58.986100 12665 heartbeater.cc:511] Master 127.24.167.62:35425 requested a full tablet report, sending...
I20260808 12:59:58.987646 12343 ts_manager.cc:194] Registered new tserver with Master: c010643f84f648dc8660dca25fde1fa9 (127.24.167.1:37687)
I20260808 12:59:58.988848 12343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:52575
Aug 08 12:59:59 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193998, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
W20260808 12:59:59.183996 12902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 12:59:59.184471 12902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 12:59:59.184660 12902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 12:59:59.196594 12902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 12:59:59.196761 12902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 12:59:59.196969 12902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 12:59:59.210995 12902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.167.20:53161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:41709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=39009
--tserver_master_addrs=127.24.167.62:35425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 12:59:59.212393 12902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 12:59:59.213732 12902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 12:59:59.229548 12908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 12:59:59.229568 12910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:59:59.230739 12902 server_base.cc:1034] running on GCE node
W20260808 12:59:59.230837 12912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 12:59:59.231781 12902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 12:59:59.232939 12902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 12:59:59.234181 12902 hybrid_clock.cc:648] HybridClock initialized: now 1786193999234102 us; error 65 us; skew 500 ppm
I20260808 12:59:59.234602 12902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 12:59:59 dist-test-slave-tk9z krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193999, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 12:59:59.238142 12902 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 12:59:59.239884 12902 webserver.cc:460] Webserver started at http://127.24.167.3:39009/ using document root <none> and password file <none>
I20260808 12:59:59.240863 12902 fs_manager.cc:359] Metadata directory not provided
I20260808 12:59:59.241032 12902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 12:59:59 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193999, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:59:59.265543 12902 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 12:59:59.269279 12927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:59.271494 12902 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 12:59:59.271685 12902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "6a4b66fc87824f7f84d3f537aadfdec8"
format_stamp: "Formatted at 2026-08-08 12:59:55 on dist-test-slave-tk9z"
server_key: "2bb8545b942276db2dbdbcf28ab4b05e"
server_key_iv: "cf4fedbf7cf77646987eabf7de56875a"
server_key_version: "kuduclusterkey@0"
I20260808 12:59:59.273126 12902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 12:59:59.314791 12902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 12:59:59.323366 12902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 12:59:59.323781 12902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 12:59:59.325227 12902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 12:59:59.327193 12902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 12:59:59.327311 12902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 12:59:59.327426 12902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 12:59:59.327504 12902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 12:59:59 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193998, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for HTTP/127.24.167.20@KRBTEST.COM
I20260808 12:59:59.449216 12902 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:41709
I20260808 12:59:59.449283 13111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:41709 every 8 connection(s)
I20260808 12:59:59.451368 12902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 12:59:59.453459 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 12902
Aug 08 12:59:59 dist-test-slave-tk9z krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786193999, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 12:59:59.516081 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.455307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.24.167.62:35425)
0808 12:59:59.494532 (+ 39225us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:59.494554 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:59.494596 (+ 42us) server_negotiation.cc:407] Connection header received
0808 12:59:59.494767 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:59.494785 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:59.494937 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:59.497427 (+ 2490us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:59.497686 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.498963 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.501079 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.501686 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.506473 (+ 4787us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.506742 (+ 269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:59.506752 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:59.506758 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:59.506803 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:59.511759 (+ 4956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:59.512732 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:59.512737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:59.512741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:59.512928 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:59.513407 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:59.513414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:59.513418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:59.513829 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:59.515291 (+ 1462us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:59.515434 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 12:59:59.515660 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":39011,"spinlock_wait_cycles":22272,"thread_start_us":38877,"threads_started":1}
I20260808 12:59:59.516924 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.457945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:35425 (local address 127.24.167.3:36717)
0808 12:59:59.481688 (+ 23743us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:59.481794 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:59.491590 (+ 9796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:59.495972 (+ 4382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:59.495985 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:59.496875 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0808 12:59:59.497810 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.497839 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.499469 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.499476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.501515 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.501530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.502840 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.502848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.503169 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.503887 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 12:59:59.503919 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 12:59:59.506642 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 12:59:59.512008 (+ 5366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:59.512018 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:59.512037 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:59.512651 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:59.513060 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 12:59:59.513065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 12:59:59.513068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 12:59:59.513359 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 12:59:59.514058 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 12:59:59.514066 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 12:59:59.514557 (+ 491us) client_negotiation.cc:769] Sending connection context
0808 12:59:59.514891 (+ 334us) client_negotiation.cc:240] Negotiation successful
0808 12:59:59.515492 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23172,"thread_start_us":25769,"threads_started":1}
I20260808 12:59:59.517836 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.456693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:36717 (local address 127.24.167.62:35425)
0808 12:59:59.495170 (+ 38477us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:59.495176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:59.495190 (+ 14us) server_negotiation.cc:407] Connection header received
0808 12:59:59.495240 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:59.495246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:59.495340 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:59.495451 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:59.498071 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.499278 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.501696 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.502350 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.502507 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.506914 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:59.506923 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:59.506927 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:59.506978 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:59.511849 (+ 4871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:59.512781 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:59.512786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:59.512789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:59.512918 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:59.513505 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:59.513513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:59.513516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:59.513829 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:59.513970 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:59.517367 (+ 3397us) server_negotiation.cc:299] Negotiation successful
0808 12:59:59.517524 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":38306,"thread_start_us":49980,"threads_started":1}
I20260808 12:59:59.519740 13114 heartbeater.cc:347] Connected to a master server at 127.24.167.62:35425
I20260808 12:59:59.520071 13114 heartbeater.cc:464] Registering TS with master...
I20260808 12:59:59.520752 13114 heartbeater.cc:511] Master 127.24.167.62:35425 requested a full tablet report, sending...
I20260808 12:59:59.522305 12343 ts_manager.cc:194] Registered new tserver with Master: 6a4b66fc87824f7f84d3f537aadfdec8 (127.24.167.3:41709)
I20260808 12:59:59.523388 12343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:36717
I20260808 12:59:59.532063 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 12:59:59.600440 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.566792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44098 (local address 127.24.167.62:35425)
0808 12:59:59.567052 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:59.567062 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:59.577549 (+ 10487us) server_negotiation.cc:407] Connection header received
0808 12:59:59.578292 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:59.578299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:59.578431 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:59.578518 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 12:59:59.581585 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.582794 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.584926 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.585579 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.585695 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.592908 (+ 7213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 12:59:59.592916 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 12:59:59.592921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 12:59:59.592971 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 12:59:59.596783 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:59.597836 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:59.597843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:59.597846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:59.598016 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 12:59:59.598534 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 12:59:59.598539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 12:59:59.598543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 12:59:59.598968 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 12:59:59.599201 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:59.599827 (+ 626us) server_negotiation.cc:299] Negotiation successful
0808 12:59:59.600083 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 12:59:59.608234 12343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 12:59:59.619501 12343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 12:59:59.686045 13143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.639084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37687 (local address 127.0.0.1:48890)
0808 12:59:59.663745 (+ 24661us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:59.663772 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:59.663977 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:59.665970 (+ 1993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:59.665982 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:59.666029 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:59:59.666696 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.666716 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.679771 (+ 13055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.679778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.682586 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.682605 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.685021 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.685028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.685248 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.685305 (+ 57us) client_negotiation.cc:769] Sending connection context
0808 12:59:59.685445 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 12:59:59.685668 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24487,"threads_started":1}
I20260808 12:59:59.688207 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.641051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.24.167.1:37687)
0808 12:59:59.664618 (+ 23567us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:59.664627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:59.664662 (+ 35us) server_negotiation.cc:407] Connection header received
0808 12:59:59.664774 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:59.664783 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:59.665201 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:59.665521 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:59:59.670596 (+ 5075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.672244 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.682880 (+ 10636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.684455 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.684585 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.684858 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:59.687613 (+ 2755us) server_negotiation.cc:299] Negotiation successful
0808 12:59:59.687815 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23371,"thread_start_us":23543,"threads_started":1}
I20260808 12:59:59.688923 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.637287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:40057 (local address 127.0.0.1:42538)
0808 12:59:59.638081 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:59.638125 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:59.638340 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:59.668544 (+ 30204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:59.668549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:59.668579 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:59:59.669567 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.669590 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.671858 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.671865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.683883 (+ 12018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.683903 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.687986 (+ 4083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.687994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.688186 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.688225 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 12:59:59.688355 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 12:59:59.688509 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":339,"thread_start_us":394,"threads_started":1}
I20260808 12:59:59.691570 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.639522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:41709 (local address 127.0.0.1:54266)
0808 12:59:59.672447 (+ 32925us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:59.672470 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:59.672708 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:59.673788 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:59.673793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:59.673822 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:59:59.674197 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.674209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.676382 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.676387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.686977 (+ 10590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.686991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.690762 (+ 3771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.690771 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.690980 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.691027 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 12:59:59.691108 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 12:59:59.691252 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29959,"thread_start_us":29440,"threads_started":1}
I20260808 12:59:59.691810 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.637357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.24.167.2:40057)
0808 12:59:59.667757 (+ 30400us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:59.667770 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:59.667812 (+ 42us) server_negotiation.cc:407] Connection header received
0808 12:59:59.667952 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:59.667965 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:59.668307 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:59.668556 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:59:59.669791 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.671678 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.684159 (+ 12481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.685654 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.685819 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.686040 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:59.691220 (+ 5180us) server_negotiation.cc:299] Negotiation successful
0808 12:59:59.691448 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30167,"thread_start_us":30035,"threads_started":1}
I20260808 12:59:59.692667 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.640196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.24.167.3:41709)
0808 12:59:59.662911 (+ 22715us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:59.662927 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:59.672581 (+ 9654us) server_negotiation.cc:407] Connection header received
0808 12:59:59.672919 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:59.672927 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:59.673194 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:59.673405 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:59:59.674442 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.676174 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.687179 (+ 11005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.688788 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.688935 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.692067 (+ 3132us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:59.692158 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 12:59:59.692322 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22432,"thread_start_us":22746,"threads_started":1}
I20260808 12:59:59.693252 12552 tablet_service.cc:1467] Processing CreateTablet for tablet 76eddb8e27514536a6d920f46123a7ae (DEFAULT_TABLE table=test-table [id=7fca3490317f4cd788b300d1b812c477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:59:59.696923 12552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76eddb8e27514536a6d920f46123a7ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:59:59.696549 12789 tablet_service.cc:1467] Processing CreateTablet for tablet 76eddb8e27514536a6d920f46123a7ae (DEFAULT_TABLE table=test-table [id=7fca3490317f4cd788b300d1b812c477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:59:59.696492 13012 tablet_service.cc:1467] Processing CreateTablet for tablet 76eddb8e27514536a6d920f46123a7ae (DEFAULT_TABLE table=test-table [id=7fca3490317f4cd788b300d1b812c477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 12:59:59.698014 13012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76eddb8e27514536a6d920f46123a7ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:59:59.698025 12789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76eddb8e27514536a6d920f46123a7ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 12:59:59.718134 12896 heartbeater.cc:503] Master 127.24.167.62:35425 was elected leader, sending a full tablet report...
I20260808 12:59:59.724319 13150 tablet_bootstrap.cc:492] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8: Bootstrap starting.
I20260808 12:59:59.729405 13151 tablet_bootstrap.cc:492] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130: Bootstrap starting.
I20260808 12:59:59.730772 13150 tablet_bootstrap.cc:654] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8: Neither blocks nor log segments found. Creating new log.
I20260808 12:59:59.732461 13150 log.cc:824] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8: Log is configured to *not* fsync() on all Append() calls
I20260808 12:59:59.742300 13154 tablet_bootstrap.cc:492] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9: Bootstrap starting.
I20260808 12:59:59.743522 13150 tablet_bootstrap.cc:492] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8: No bootstrap required, opened a new log
I20260808 12:59:59.743968 13150 ts_tablet_manager.cc:1397] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.004s
I20260808 12:59:59.744488 13151 tablet_bootstrap.cc:654] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130: Neither blocks nor log segments found. Creating new log.
I20260808 12:59:59.746028 13151 log.cc:824] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130: Log is configured to *not* fsync() on all Append() calls
I20260808 12:59:59.746896 13154 tablet_bootstrap.cc:654] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9: Neither blocks nor log segments found. Creating new log.
I20260808 12:59:59.748314 13154 log.cc:824] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9: Log is configured to *not* fsync() on all Append() calls
I20260808 12:59:59.755625 13150 raft_consensus.cc:348] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eddfdf820ac84ab68137ba8fcd57f130" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40057 } } peers { permanent_uuid: "c010643f84f648dc8660dca25fde1fa9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37687 } } peers { permanent_uuid: "6a4b66fc87824f7f84d3f537aadfdec8" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 41709 } }
I20260808 12:59:59.756147 13150 raft_consensus.cc:374] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:59:59.756269 13150 raft_consensus.cc:729] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4b66fc87824f7f84d3f537aadfdec8, State: Initialized, Role: FOLLOWER
I20260808 12:59:59.756983 13150 consensus_queue.cc:260] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eddfdf820ac84ab68137ba8fcd57f130" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40057 } } peers { permanent_uuid: "c010643f84f648dc8660dca25fde1fa9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37687 } } peers { permanent_uuid: "6a4b66fc87824f7f84d3f537aadfdec8" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 41709 } }
I20260808 12:59:59.760175 13114 heartbeater.cc:503] Master 127.24.167.62:35425 was elected leader, sending a full tablet report...
I20260808 12:59:59.761230 13154 tablet_bootstrap.cc:492] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9: No bootstrap required, opened a new log
I20260808 12:59:59.761242 13151 tablet_bootstrap.cc:492] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130: No bootstrap required, opened a new log
I20260808 12:59:59.761556 13151 ts_tablet_manager.cc:1397] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.009s
I20260808 12:59:59.762836 13150 ts_tablet_manager.cc:1428] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8: Time spent starting tablet: real 0.019s user 0.011s sys 0.004s
I20260808 12:59:59.763897 13154 ts_tablet_manager.cc:1397] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20260808 12:59:59.772018 13151 raft_consensus.cc:348] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eddfdf820ac84ab68137ba8fcd57f130" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40057 } } peers { permanent_uuid: "c010643f84f648dc8660dca25fde1fa9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37687 } } peers { permanent_uuid: "6a4b66fc87824f7f84d3f537aadfdec8" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 41709 } }
I20260808 12:59:59.772567 13151 raft_consensus.cc:374] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:59:59.772699 13151 raft_consensus.cc:729] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eddfdf820ac84ab68137ba8fcd57f130, State: Initialized, Role: FOLLOWER
I20260808 12:59:59.773402 13151 consensus_queue.cc:260] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eddfdf820ac84ab68137ba8fcd57f130" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40057 } } peers { permanent_uuid: "c010643f84f648dc8660dca25fde1fa9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37687 } } peers { permanent_uuid: "6a4b66fc87824f7f84d3f537aadfdec8" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 41709 } }
I20260808 12:59:59.775220 13151 ts_tablet_manager.cc:1428] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260808 12:59:59.776073 13154 raft_consensus.cc:348] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eddfdf820ac84ab68137ba8fcd57f130" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40057 } } peers { permanent_uuid: "c010643f84f648dc8660dca25fde1fa9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37687 } } peers { permanent_uuid: "6a4b66fc87824f7f84d3f537aadfdec8" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 41709 } }
I20260808 12:59:59.776588 13154 raft_consensus.cc:374] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 12:59:59.776705 13154 raft_consensus.cc:729] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c010643f84f648dc8660dca25fde1fa9, State: Initialized, Role: FOLLOWER
I20260808 12:59:59.777432 13154 consensus_queue.cc:260] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eddfdf820ac84ab68137ba8fcd57f130" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40057 } } peers { permanent_uuid: "c010643f84f648dc8660dca25fde1fa9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37687 } } peers { permanent_uuid: "6a4b66fc87824f7f84d3f537aadfdec8" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 41709 } }
I20260808 12:59:59.778861 12665 heartbeater.cc:503] Master 127.24.167.62:35425 was elected leader, sending a full tablet report...
I20260808 12:59:59.779574 13154 ts_tablet_manager.cc:1428] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260808 12:59:59.854022 13169 raft_consensus.cc:482] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:59:59.854462 13169 raft_consensus.cc:504] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eddfdf820ac84ab68137ba8fcd57f130" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40057 } } peers { permanent_uuid: "c010643f84f648dc8660dca25fde1fa9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37687 } } peers { permanent_uuid: "6a4b66fc87824f7f84d3f537aadfdec8" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 41709 } }
I20260808 12:59:59.856410 13169 leader_election.cc:288] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eddfdf820ac84ab68137ba8fcd57f130 (127.24.167.2:40057), 6a4b66fc87824f7f84d3f537aadfdec8 (127.24.167.3:41709)
I20260808 12:59:59.863623 13164 raft_consensus.cc:482] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 12:59:59.863936 13164 raft_consensus.cc:504] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eddfdf820ac84ab68137ba8fcd57f130" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40057 } } peers { permanent_uuid: "c010643f84f648dc8660dca25fde1fa9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37687 } } peers { permanent_uuid: "6a4b66fc87824f7f84d3f537aadfdec8" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 41709 } }
I20260808 12:59:59.865139 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.857282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:45853 (local address 127.24.167.3:41709)
0808 12:59:59.857474 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:59.857480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:59.857783 (+ 303us) server_negotiation.cc:407] Connection header received
0808 12:59:59.858088 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:59.858092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:59.858154 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:59.858255 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:59:59.859131 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.860143 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.862329 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.863494 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.863665 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.863750 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:59.864305 (+ 555us) server_negotiation.cc:299] Negotiation successful
0808 12:59:59.864456 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 12:59:59.865288 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.857725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:50023 (local address 127.24.167.2:40057)
0808 12:59:59.857927 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:59.857932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:59.857951 (+ 19us) server_negotiation.cc:407] Connection header received
0808 12:59:59.858027 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:59.858033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:59.858108 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:59.858214 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:59:59.858915 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.860034 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.862827 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.864088 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.864284 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.864355 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:59.864896 (+ 541us) server_negotiation.cc:299] Negotiation successful
0808 12:59:59.865037 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 12:59:59.865921 13170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.856724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:41709 (local address 127.24.167.1:45853)
0808 12:59:59.857678 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:59.857693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:59.857887 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:59.858440 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:59.858444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:59.858487 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:59:59.858957 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.858969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.860407 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.860416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.862161 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.862177 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.863905 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.863911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.864084 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.864119 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 12:59:59.865326 (+ 1207us) client_negotiation.cc:240] Negotiation successful
0808 12:59:59.865473 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":1,"thread_start_us":361,"threads_started":1}
I20260808 12:59:59.865957 13171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.856648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:40057 (local address 127.24.167.1:50023)
0808 12:59:59.857477 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:59.857506 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:59.857679 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:59.858255 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:59.858260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:59.858286 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:59:59.858743 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.858754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.860196 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.860202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.862658 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.862674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.864499 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.864504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.864678 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.864710 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 12:59:59.865443 (+ 733us) client_negotiation.cc:240] Negotiation successful
0808 12:59:59.865551 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":54,"spinlock_wait_cycles":19840,"thread_start_us":1134,"threads_started":1}
I20260808 12:59:59.866000 13164 leader_election.cc:288] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eddfdf820ac84ab68137ba8fcd57f130 (127.24.167.2:40057), c010643f84f648dc8660dca25fde1fa9 (127.24.167.1:37687)
I20260808 12:59:59.867376 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76eddb8e27514536a6d920f46123a7ae" candidate_uuid: "c010643f84f648dc8660dca25fde1fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4b66fc87824f7f84d3f537aadfdec8" is_pre_election: true
I20260808 12:59:59.867895 13039 raft_consensus.cc:2454] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c010643f84f648dc8660dca25fde1fa9 in term 0.
I20260808 12:59:59.868117 12821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76eddb8e27514536a6d920f46123a7ae" candidate_uuid: "c010643f84f648dc8660dca25fde1fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eddfdf820ac84ab68137ba8fcd57f130" is_pre_election: true
I20260808 12:59:59.868822 12821 raft_consensus.cc:2454] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c010643f84f648dc8660dca25fde1fa9 in term 0.
I20260808 12:59:59.868988 12482 leader_election.cc:302] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a4b66fc87824f7f84d3f537aadfdec8, c010643f84f648dc8660dca25fde1fa9; no voters:
I20260808 12:59:59.869699 13169 raft_consensus.cc:2781] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 12:59:59.869866 13169 raft_consensus.cc:482] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 12:59:59.869978 13169 raft_consensus.cc:3037] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:59:59.873849 13169 raft_consensus.cc:504] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eddfdf820ac84ab68137ba8fcd57f130" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40057 } } peers { permanent_uuid: "c010643f84f648dc8660dca25fde1fa9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37687 } } peers { permanent_uuid: "6a4b66fc87824f7f84d3f537aadfdec8" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 41709 } }
I20260808 12:59:59.875242 12821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76eddb8e27514536a6d920f46123a7ae" candidate_uuid: "c010643f84f648dc8660dca25fde1fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eddfdf820ac84ab68137ba8fcd57f130"
I20260808 12:59:59.875582 12821 raft_consensus.cc:3037] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:59:59.876794 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.867305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37687 (local address 127.24.167.3:45059)
0808 12:59:59.868088 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:59.868114 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:59.868251 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:59.869229 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:59.869235 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:59.869265 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:59:59.869462 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.869474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.871773 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.871779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.874092 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.874108 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.875982 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.875989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.876198 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.876246 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 12:59:59.876348 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 12:59:59.876486 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":1}
I20260808 12:59:59.877363 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.866734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:45059 (local address 127.24.167.1:37687)
0808 12:59:59.866955 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:59.866960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:59.868739 (+ 1779us) server_negotiation.cc:407] Connection header received
0808 12:59:59.868839 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:59.868847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:59.868938 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:59.869041 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:59:59.870108 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.871581 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.874360 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.875603 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.875767 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.875864 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:59.876997 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0808 12:59:59.877134 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 12:59:59.877774 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.868645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:47641 (local address 127.24.167.2:40057)
0808 12:59:59.870237 (+ 1592us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:59.870243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:59.870260 (+ 17us) server_negotiation.cc:407] Connection header received
0808 12:59:59.870344 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:59.870348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:59.870439 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:59.870519 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 12:59:59.871494 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.872791 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.875358 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.876574 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.876726 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.876803 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:59.877325 (+ 522us) server_negotiation.cc:299] Negotiation successful
0808 12:59:59.877470 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1472}
I20260808 12:59:59.878443 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.867392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:40057 (local address 127.24.167.3:47641)
0808 12:59:59.869790 (+ 2398us) negotiation.cc:107] Waiting for socket to connect
0808 12:59:59.869808 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 12:59:59.869911 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 12:59:59.870749 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 12:59:59.870754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 12:59:59.870777 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 12:59:59.871204 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.871217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.873115 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.873120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.875152 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 12:59:59.875167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.876984 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.876989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 12:59:59.877141 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.877170 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 12:59:59.877955 (+ 785us) client_negotiation.cc:240] Negotiation successful
0808 12:59:59.878095 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2292,"thread_start_us":2387,"threads_started":1}
I20260808 12:59:59.878470 12592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76eddb8e27514536a6d920f46123a7ae" candidate_uuid: "6a4b66fc87824f7f84d3f537aadfdec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c010643f84f648dc8660dca25fde1fa9" is_pre_election: true
I20260808 12:59:59.878706 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76eddb8e27514536a6d920f46123a7ae" candidate_uuid: "c010643f84f648dc8660dca25fde1fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4b66fc87824f7f84d3f537aadfdec8"
I20260808 12:59:59.878998 12592 raft_consensus.cc:2379] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a4b66fc87824f7f84d3f537aadfdec8 in current term 1: Already voted for candidate c010643f84f648dc8660dca25fde1fa9 in this term.
I20260808 12:59:59.878966 13039 raft_consensus.cc:3037] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 12:59:59.878964 13169 leader_election.cc:288] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [CANDIDATE]: Term 1 election: Requested vote from peers eddfdf820ac84ab68137ba8fcd57f130 (127.24.167.2:40057), 6a4b66fc87824f7f84d3f537aadfdec8 (127.24.167.3:41709)
I20260808 12:59:59.880755 12820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76eddb8e27514536a6d920f46123a7ae" candidate_uuid: "6a4b66fc87824f7f84d3f537aadfdec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eddfdf820ac84ab68137ba8fcd57f130" is_pre_election: true
I20260808 12:59:59.881871 12930 leader_election.cc:302] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a4b66fc87824f7f84d3f537aadfdec8; no voters: c010643f84f648dc8660dca25fde1fa9, eddfdf820ac84ab68137ba8fcd57f130
I20260808 12:59:59.882282 12821 raft_consensus.cc:2454] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c010643f84f648dc8660dca25fde1fa9 in term 1.
I20260808 12:59:59.882977 12479 leader_election.cc:302] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c010643f84f648dc8660dca25fde1fa9, eddfdf820ac84ab68137ba8fcd57f130; no voters:
I20260808 12:59:59.883287 13169 raft_consensus.cc:2781] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 12:59:59.883755 13169 raft_consensus.cc:686] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [term 1 LEADER]: Becoming Leader. State: Replica: c010643f84f648dc8660dca25fde1fa9, State: Running, Role: LEADER
I20260808 12:59:59.884256 13039 raft_consensus.cc:2454] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c010643f84f648dc8660dca25fde1fa9 in term 1.
I20260808 12:59:59.884330 13169 consensus_queue.cc:237] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eddfdf820ac84ab68137ba8fcd57f130" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40057 } } peers { permanent_uuid: "c010643f84f648dc8660dca25fde1fa9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37687 } } peers { permanent_uuid: "6a4b66fc87824f7f84d3f537aadfdec8" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 41709 } }
I20260808 12:59:59.884590 13164 raft_consensus.cc:2726] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 12:59:59.889752 12343 catalog_manager.cc:5495] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c010643f84f648dc8660dca25fde1fa9 (127.24.167.1). New cstate: current_term: 1 leader_uuid: "c010643f84f648dc8660dca25fde1fa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eddfdf820ac84ab68137ba8fcd57f130" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c010643f84f648dc8660dca25fde1fa9" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 37687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a4b66fc87824f7f84d3f537aadfdec8" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 41709 } health_report { overall_health: UNKNOWN } } }
W20260808 12:59:59.933898 12897 tablet.cc:2369] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 12:59:59.937897 12668 tablet.cc:2369] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 12:59:59.954769 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 12:59:59.948234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.24.167.1:37687)
0808 12:59:59.948413 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 12:59:59.948417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 12:59:59.948435 (+ 18us) server_negotiation.cc:407] Connection header received
0808 12:59:59.948711 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 12:59:59.948715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 12:59:59.948812 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 12:59:59.948968 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 12:59:59.949425 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.950166 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.952275 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 12:59:59.952699 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 12:59:59.952801 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 12:59:59.953480 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 12:59:59.954028 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 12:59:59.954138 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 12:59:59.954295 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 12:59:59.954413 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 12:59:59.963173 12821 raft_consensus.cc:1261] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130 [term 1 FOLLOWER]: Refusing update from remote peer c010643f84f648dc8660dca25fde1fa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:59:59.963224 13039 raft_consensus.cc:1261] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8 [term 1 FOLLOWER]: Refusing update from remote peer c010643f84f648dc8660dca25fde1fa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 12:59:59.964110 13169 consensus_queue.cc:1035] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a4b66fc87824f7f84d3f537aadfdec8" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:59:59.964813 13173 consensus_queue.cc:1035] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eddfdf820ac84ab68137ba8fcd57f130" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 12:59:59.996028 13180 mvcc.cc:205] Tried to move back new op lower bound from 7316250623838101504 to 7316250623531466752. Current Snapshot: MvccSnapshot[applied={T|T < 7316250623838101504}]
I20260808 13:00:00.005728 12343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:00:00.006047 12343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:00:00.010113 12343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a2e12a24b5244b6b1b14775e508f061: Sending DeleteTablet for 3 replicas of tablet 76eddb8e27514536a6d920f46123a7ae
I20260808 13:00:00.011514 12552 tablet_service.cc:1514] Processing DeleteTablet for tablet 76eddb8e27514536a6d920f46123a7ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:00:00 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:48890
I20260808 13:00:00.011512 12789 tablet_service.cc:1514] Processing DeleteTablet for tablet 76eddb8e27514536a6d920f46123a7ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:00:00 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:42538
I20260808 13:00:00.013186 13012 tablet_service.cc:1514] Processing DeleteTablet for tablet 76eddb8e27514536a6d920f46123a7ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:00:00 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:54266
I20260808 13:00:00.023481 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 12399
I20260808 13:00:00.042039 13191 tablet_replica.cc:330] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8: stopping tablet replica
I20260808 13:00:00.042830 13191 raft_consensus.cc:2229] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:00:00.043694 13191 raft_consensus.cc:2258] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:00:00.046361 13191 ts_tablet_manager.cc:1905] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:00:00.052500 13190 tablet_replica.cc:330] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130: stopping tablet replica
I20260808 13:00:00.053122 13189 tablet_replica.cc:330] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9: stopping tablet replica
I20260808 13:00:00.053784 13189 raft_consensus.cc:2229] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 13:00:00.054069 13191 ts_tablet_manager.cc:1918] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:00:00.054246 13191 log.cc:1196] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/76eddb8e27514536a6d920f46123a7ae
I20260808 13:00:00.054886 13191 ts_tablet_manager.cc:1939] T 76eddb8e27514536a6d920f46123a7ae P 6a4b66fc87824f7f84d3f537aadfdec8: Deleting consensus metadata
I20260808 13:00:00.056721 13190 raft_consensus.cc:2229] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:00:00.057160 12318 catalog_manager.cc:4841] TS 6a4b66fc87824f7f84d3f537aadfdec8 (127.24.167.3:41709): tablet 76eddb8e27514536a6d920f46123a7ae (table test-table [id=7fca3490317f4cd788b300d1b812c477]) successfully deleted
I20260808 13:00:00.057834 13190 raft_consensus.cc:2258] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:00:00.058224 13189 raft_consensus.cc:2258] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:00:00.062348 13190 ts_tablet_manager.cc:1905] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:00:00.062356 13189 ts_tablet_manager.cc:1905] T 76eddb8e27514536a6d920f46123a7ae P c010643f84f648dc8660dca25fde1fa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:00:00.068513 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:00.025381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.24.167.1:37687)
0808 13:00:00.025668 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:00.025677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:00.045094 (+ 19417us) server_negotiation.cc:407] Connection header received
0808 13:00:00.045649 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:00.045656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:00.045752 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:00.045871 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:00.048113 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:00.049302 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:00.054310 (+ 5008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:00.055216 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:00.055342 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:00.060575 (+ 5233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:00.060585 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:00.060602 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:00.060654 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:00.064941 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:00.065917 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:00.065926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:00.065930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:00.066136 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:00.066623 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:00.066628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:00.066631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:00.067248 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:00.067587 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:00.067900 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 13:00:00.068163 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 13:00:00.069418 12657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:00.071949 13190 ts_tablet_manager.cc:1918] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:00:00.072139 13190 log.cc:1196] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/76eddb8e27514536a6d920f46123a7ae
I20260808 13:00:00.072746 13190 ts_tablet_manager.cc:1939] T 76eddb8e27514536a6d920f46123a7ae P eddfdf820ac84ab68137ba8fcd57f130: Deleting consensus metadata
I20260808 13:00:00.074710 12315 catalog_manager.cc:4841] TS eddfdf820ac84ab68137ba8fcd57f130 (127.24.167.2:40057): tablet 76eddb8e27514536a6d920f46123a7ae (table test-table [id=7fca3490317f4cd788b300d1b812c477]) successfully deleted
I20260808 13:00:00.177361 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 12399
I20260808 13:00:00.203372 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 12671
W20260808 13:00:00.203397 12316 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.167.1:37687 (error 108)
I20260808 13:00:00.205467 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:00.204717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37687 (local address 127.0.0.1:48908)
0808 13:00:00.205184 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:00.205287 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":215,"threads_started":1}
W20260808 13:00:00.205685 12316 catalog_manager.cc:4584] TS c010643f84f648dc8660dca25fde1fa9 (127.24.167.1:37687): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76eddb8e27514536a6d920f46123a7ae: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37687: connect: Connection refused (error 111)
I20260808 13:00:00.217129 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:00.204205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.24.167.2:40057)
0808 13:00:00.204436 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:00.204443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:00.204540 (+ 97us) server_negotiation.cc:407] Connection header received
0808 13:00:00.204736 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:00.204740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:00.204805 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:00.204973 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:00.206524 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:00.207256 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:00.208448 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:00.208829 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:00.208918 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:00.211480 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:00.211492 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:00.211505 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:00.211538 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:00.214628 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:00.215304 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:00.215312 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:00.215314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:00.215462 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:00.215823 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:00.215826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:00.215829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:00.216188 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:00.216437 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:00.216693 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 13:00:00.216867 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 13:00:00.217828 12885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:00.259573 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:00.258964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37687 (local address 127.0.0.1:48916)
0808 13:00:00.259266 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:00.259390 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 13:00:00.260764 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:00.260335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37687 (local address 127.0.0.1:48918)
0808 13:00:00.260505 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:00.260589 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 13:00:00.310240 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 12671
I20260808 13:00:00.321466 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:00.320814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37687 (local address 127.0.0.1:48920)
0808 13:00:00.321119 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:00.321249 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 13:00:00.322768 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:00.322247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37687 (local address 127.0.0.1:48926)
0808 13:00:00.322471 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:00.322570 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 13:00:00.337157 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 12902
I20260808 13:00:00.350761 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:00.337947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54270 (local address 127.24.167.3:41709)
0808 13:00:00.338124 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:00.338128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:00.338147 (+ 19us) server_negotiation.cc:407] Connection header received
0808 13:00:00.338291 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:00.338295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:00.338358 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:00.338510 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:00.339861 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:00.340630 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:00.341959 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:00.342331 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:00.342504 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:00.345043 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:00.345059 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:00.345077 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:00.345107 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:00.348036 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:00.348659 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:00.348666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:00.348668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:00.348800 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:00.349146 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:00.349149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:00.349151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:00.349502 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:00.349708 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:00.350082 (+ 374us) server_negotiation.cc:299] Negotiation successful
0808 13:00:00.350234 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 13:00:00.351423 13103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:00.404156 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:00.403550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37687 (local address 127.0.0.1:48930)
0808 13:00:00.403828 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:00.403958 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 13:00:00.405402 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:00.404900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37687 (local address 127.0.0.1:48942)
0808 13:00:00.405141 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:00.405229 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260808 13:00:00.457569 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 12902
I20260808 13:00:00.486137 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 12284
I20260808 13:00:00.487571 12390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:00.601059 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:00.600334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:37687 (local address 127.0.0.1:48950)
0808 13:00:00.600660 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:00.600822 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 13:00:00.601157 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 12284
I20260808 13:00:00.630990 25244 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 13:00:00.976536 25244 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 13:00:00.976743 25244 mini_ranger.cc:66] Stopping Ranger...
I20260808 13:00:01.332576 25244 mini_ranger.cc:68] Stopped Ranger
2026-08-08 13:00:01.332 UTC [10799] LOG: received smart shutdown request
2026-08-08 13:00:01.336 UTC [10799] LOG: background worker "logical replication launcher" (PID 10811) exited with exit code 1
2026-08-08 13:00:01.336 UTC [10804] LOG: shutting down
2026-08-08 13:00:01.477 UTC [10799] LOG: database system is shut down
2026-08-08T13:00:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53957 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:00:01 dist-test-slave-tk9z krb5kdc[13216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:00:01 dist-test-slave-tk9z krb5kdc[13216](info): set up 2 sockets
Aug 08 13:00:01 dist-test-slave-tk9z krb5kdc[13216](info): commencing operation
krb5kdc: starting...
W20260808 13:00:02.946952 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.292s user 0.006s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:00:03 dist-test-slave-tk9z krb5kdc[13216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:00:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:00:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 13:00:03.236372 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:41633
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:41633,127.24.167.61:46515,127.24.167.60:35005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:00:03.586158 13238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:03.586536 13238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:03.586652 13238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:03.595012 13238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:00:03.595146 13238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:03.595199 13238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:03.595260 13238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:00:03.595317 13238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:00:03.606866 13238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:41633,127.24.167.61:46515,127.24.167.60:35005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:41633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:03.608374 13238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:03.609758 13238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:03.623417 13254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:03.623677 13251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:03.623672 13250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:03.624903 13238 server_base.cc:1034] running on GCE node
I20260808 13:00:03.625916 13238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:03.627717 13238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:03.628994 13238 hybrid_clock.cc:648] HybridClock initialized: now 1786194003628943 us; error 48 us; skew 500 ppm
I20260808 13:00:03.629437 13238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:03 dist-test-slave-tk9z krb5kdc[13216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194003, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:03.633025 13238 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:00:03.634773 13238 webserver.cc:460] Webserver started at http://127.24.167.62:42775/ using document root <none> and password file <none>
I20260808 13:00:03.635450 13238 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:03.635553 13238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:03.635815 13238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:00:03.639004 13238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "5f4b92de695b4754b6c39436569d0f5c"
format_stamp: "Formatted at 2026-08-08 13:00:03 on dist-test-slave-tk9z"
server_key: "439074c6482df8536ff3650baed0c33a"
server_key_iv: "df40da675222499a33926f0197ecd948"
server_key_version: "encryptionkey@0"
I20260808 13:00:03.639784 13238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "5f4b92de695b4754b6c39436569d0f5c"
format_stamp: "Formatted at 2026-08-08 13:00:03 on dist-test-slave-tk9z"
server_key: "439074c6482df8536ff3650baed0c33a"
server_key_iv: "df40da675222499a33926f0197ecd948"
server_key_version: "encryptionkey@0"
I20260808 13:00:03.645054 13238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 13:00:03.648555 13262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:03.650031 13238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 13:00:03.650179 13238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "5f4b92de695b4754b6c39436569d0f5c"
format_stamp: "Formatted at 2026-08-08 13:00:03 on dist-test-slave-tk9z"
server_key: "439074c6482df8536ff3650baed0c33a"
server_key_iv: "df40da675222499a33926f0197ecd948"
server_key_version: "encryptionkey@0"
I20260808 13:00:03.650465 13238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:03.666817 13238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:03.671212 13238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:03.671476 13238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:03.691833 13238 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:41633
I20260808 13:00:03.691882 13348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:41633 every 8 connection(s)
I20260808 13:00:03.693292 13238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:00:03.694536 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 13238
I20260808 13:00:03.694800 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:00:03.695693 25244 external_mini_cluster.cc:1428] Setting key 69ba5eec6207d27945d94f2184fae910
I20260808 13:00:03.698343 13349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:00:03.708199 13349 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.167.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.61@KRBTEST.COM" created.
Aug 08 13:00:03 dist-test-slave-tk9z krb5kdc[13216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194003, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:00:03.736122 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:03.711566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52374 (local address 127.24.167.62:41633)
0808 13:00:03.713435 (+ 1869us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:03.713446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:03.713523 (+ 77us) server_negotiation.cc:407] Connection header received
0808 13:00:03.715237 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:03.715248 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:03.715609 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:03.715848 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:03.717169 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:03.719280 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:03.720844 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:03.721853 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:03.722085 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:03.728555 (+ 6470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:03.728564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:03.728579 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:03.728613 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:03.731623 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:03.732631 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:03.732639 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:03.732641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:03.732798 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:03.733271 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:03.733275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:03.733277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:03.733817 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:03.733999 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:03.734716 (+ 717us) server_negotiation.cc:299] Negotiation successful
0808 13:00:03.735029 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":1616,"spinlock_wait_cycles":26368,"thread_start_us":1397,"threads_started":1}
I20260808 13:00:03.736171 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:03.711572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41633 (local address 127.0.0.1:52374)
0808 13:00:03.712712 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:03.712776 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:03.714834 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:03.715909 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:03.715914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:03.716576 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:03.716936 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:03.716963 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:03.719485 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:03.719490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:03.720674 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:03.720687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:03.722159 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:03.722165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:03.722334 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:03.723155 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:03.723179 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:03.728263 (+ 5084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:03.731779 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:03.731796 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:03.731812 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:03.732489 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:03.732917 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:03.732926 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:03.732933 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:03.733159 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:03.733974 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:03.733985 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:03.734270 (+ 285us) client_negotiation.cc:769] Sending connection context
0808 13:00:03.734617 (+ 347us) client_negotiation.cc:240] Negotiation successful
0808 13:00:03.735230 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":87,"thread_start_us":506,"threads_started":1}
I20260808 13:00:03.739567 13349 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } has no permanent_uuid. Determining permanent_uuid...
I20260808 13:00:03.740921 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:03.740285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:46515 (local address 127.0.0.1:37188)
0808 13:00:03.740634 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:03.740784 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
W20260808 13:00:03.741230 13267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
I20260808 13:00:03.743184 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:03.742744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:46515 (local address 127.0.0.1:37202)
0808 13:00:03.742966 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:03.743039 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260808 13:00:03.743433 13349 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.61:46515: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.61.keytab.
Entry for principal kudu/127.24.167.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.61.keytab.
I20260808 13:00:03.761739 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.61:46515
--webserver_interface=127.24.167.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.61.keytab
--principal=kudu/127.24.167.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:41633,127.24.167.61:46515,127.24.167.60:35005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 13:00:03.792834 13349 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } attempt: 1
I20260808 13:00:03.794167 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:03.793669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:46515 (local address 127.0.0.1:37204)
0808 13:00:03.793886 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:03.794021 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 13:00:03.795347 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:03.794979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:46515 (local address 127.0.0.1:37218)
0808 13:00:03.795134 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:03.795212 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 13:00:03.795547 13349 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.61:46515: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
I20260808 13:00:03.863786 13349 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } attempt: 2
I20260808 13:00:03.865168 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:03.864622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:46515 (local address 127.0.0.1:37230)
0808 13:00:03.864899 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:03.865014 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 13:00:03.866276 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:03.865869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:46515 (local address 127.0.0.1:37236)
0808 13:00:03.866045 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:03.866129 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260808 13:00:03.866545 13349 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.61:46515: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
I20260808 13:00:03.957793 13349 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } attempt: 3
I20260808 13:00:03.959054 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:03.958589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:46515 (local address 127.0.0.1:37250)
0808 13:00:03.958801 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:03.958917 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 13:00:03.960179 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:03.959795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:46515 (local address 127.0.0.1:37260)
0808 13:00:03.959965 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:03.960038 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 13:00:03.960397 13349 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.61:46515: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
I20260808 13:00:04.103636 13349 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } attempt: 4
I20260808 13:00:04.104930 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.104354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:46515 (local address 127.0.0.1:37274)
0808 13:00:04.104694 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.104785 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260808 13:00:04.105930 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.105559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:46515 (local address 127.0.0.1:37284)
0808 13:00:04.105702 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.105797 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 13:00:04.106124 13349 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.61:46515: Network error: Client connection negotiation failed: client connection to 127.24.167.61:46515: connect: Connection refused (error 111)
W20260808 13:00:04.107622 13360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:04.107975 13360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:04.108134 13360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:04.115747 13360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:00:04.115898 13360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:04.116005 13360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:04.116109 13360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:00:04.116191 13360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:00:04.126227 13360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:41633,127.24.167.61:46515,127.24.167.60:35005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.61.keytab
--principal=kudu/127.24.167.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.61:46515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.167.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:04.127645 13360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:04.128952 13360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:04.141332 13370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:04.141289 13367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:04.141368 13368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:04.142333 13360 server_base.cc:1034] running on GCE node
I20260808 13:00:04.143203 13360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:04.144608 13360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:04.145807 13360 hybrid_clock.cc:648] HybridClock initialized: now 1786194004145771 us; error 46 us; skew 500 ppm
I20260808 13:00:04.146152 13360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:04 dist-test-slave-tk9z krb5kdc[13216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194004, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:04.149708 13360 init.cc:377] Logged in from keytab as kudu/127.24.167.61@KRBTEST.COM (short username kudu)
I20260808 13:00:04.151160 13360 webserver.cc:460] Webserver started at http://127.24.167.61:41355/ using document root <none> and password file <none>
I20260808 13:00:04.151844 13360 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:04.151991 13360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:04.152303 13360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:00:04.155378 13360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/data/instance:
uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407"
format_stamp: "Formatted at 2026-08-08 13:00:04 on dist-test-slave-tk9z"
server_key: "c5d89fe2b9fdfd8046e219d2aadb5138"
server_key_iv: "e52c1074b88ff162ba5388be9277e8f7"
server_key_version: "encryptionkey@0"
I20260808 13:00:04.156279 13360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/wal/instance:
uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407"
format_stamp: "Formatted at 2026-08-08 13:00:04 on dist-test-slave-tk9z"
server_key: "c5d89fe2b9fdfd8046e219d2aadb5138"
server_key_iv: "e52c1074b88ff162ba5388be9277e8f7"
server_key_version: "encryptionkey@0"
I20260808 13:00:04.161320 13360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 13:00:04.164669 13383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:04.165942 13360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 13:00:04.166131 13360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/wal
uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407"
format_stamp: "Formatted at 2026-08-08 13:00:04 on dist-test-slave-tk9z"
server_key: "c5d89fe2b9fdfd8046e219d2aadb5138"
server_key_iv: "e52c1074b88ff162ba5388be9277e8f7"
server_key_version: "encryptionkey@0"
I20260808 13:00:04.166333 13360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:04.178759 13360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:04.183707 13360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:04.184016 13360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:04.202255 13360 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.61:46515
I20260808 13:00:04.202275 13467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.61:46515 every 8 connection(s)
I20260808 13:00:04.203761 13360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/data/info.pb
I20260808 13:00:04.207680 13468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:00:04.209510 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 13360
I20260808 13:00:04.209714 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-1/wal/instance
I20260808 13:00:04.210206 25244 external_mini_cluster.cc:1428] Setting key eff2b5c893d7d7aa6cc833f880f17b12
I20260808 13:00:04.214226 13468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 13:00:04 dist-test-slave-tk9z krb5kdc[13216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194004, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.61@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.167.60@KRBTEST.COM; defaulting to no policy
I20260808 13:00:04.235301 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.217108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.24.167.62:41633)
0808 13:00:04.217321 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:04.217332 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:04.218593 (+ 1261us) server_negotiation.cc:407] Connection header received
0808 13:00:04.220505 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:04.220511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:04.220609 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:04.220730 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:04.222250 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.223007 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.224552 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.224984 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.225126 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.228984 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:04.228989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:04.228992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:04.229024 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:04.231967 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.232741 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.232748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.232751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.232880 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.233316 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.233320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.233325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.233628 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:04.233801 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:04.234675 (+ 874us) server_negotiation.cc:299] Negotiation successful
0808 13:00:04.234973 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.60@KRBTEST.COM" created.
I20260808 13:00:04.236456 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.217544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41633 (local address 127.0.0.1:52382)
0808 13:00:04.218353 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.218460 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:04.220214 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:04.220979 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:04.220988 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:04.221502 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:04.222069 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.222086 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.223180 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.223184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.224379 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.224392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.225221 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.225229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.225456 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.226178 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:04.226202 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:04.228746 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:04.232139 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.232148 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.232165 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.232615 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.233003 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.233008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.233012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.233199 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.233777 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:04.233785 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:04.234213 (+ 428us) client_negotiation.cc:769] Sending connection context
0808 13:00:04.234582 (+ 369us) client_negotiation.cc:240] Negotiation successful
0808 13:00:04.235111 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":364,"threads_started":1}
I20260808 13:00:04.238426 13468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 13:00:04 dist-test-slave-tk9z krb5kdc[13216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194004, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.61@KRBTEST.COM for kudu/127.24.167.61@KRBTEST.COM
I20260808 13:00:04.256031 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.239649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.24.167.61:46515)
0808 13:00:04.240382 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:04.240393 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:04.240427 (+ 34us) server_negotiation.cc:407] Connection header received
0808 13:00:04.240505 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:04.240515 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:04.240754 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:04.240920 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:04.241620 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.243439 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.245150 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.246114 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.246258 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.249983 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:04.249995 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:04.250009 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:04.250037 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:04.252924 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.253604 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.253617 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.253618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.253780 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.254290 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.254295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.254297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.254745 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:04.255010 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:04.255397 (+ 387us) server_negotiation.cc:299] Negotiation successful
0808 13:00:04.255700 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":535,"thread_start_us":367,"threads_started":1}
I20260808 13:00:04.256120 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.239156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:46515 (local address 127.0.0.1:37286)
0808 13:00:04.239405 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.239426 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:04.239551 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:04.240929 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:04.240933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:04.241307 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:04.241455 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.241464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.243609 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.243613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.244994 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.245010 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.246269 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.246273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.246525 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.247155 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:04.247173 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:04.249746 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:04.253071 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.253078 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.253084 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.253465 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.253924 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.253931 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.253937 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.254146 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.254899 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:04.254907 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:04.255083 (+ 176us) client_negotiation.cc:769] Sending connection context
0808 13:00:04.255248 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 13:00:04.255598 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 13:00:04.258560 13468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } has no permanent_uuid. Determining permanent_uuid...
I20260808 13:00:04.259886 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.259437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:43920)
0808 13:00:04.259622 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.259738 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 13:00:04.260317 13390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
I20260808 13:00:04.261991 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.261677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:43936)
0808 13:00:04.261805 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.261886 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 13:00:04.262324 13468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:35005: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.60.keytab.
Entry for principal kudu/127.24.167.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.60.keytab.
I20260808 13:00:04.271889 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.60:35005
--webserver_interface=127.24.167.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.60.keytab
--principal=kudu/127.24.167.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:41633,127.24.167.61:46515,127.24.167.60:35005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 13:00:04.311779 13468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } attempt: 1
I20260808 13:00:04.313172 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.312644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:43946)
0808 13:00:04.312880 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.313002 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260808 13:00:04.314273 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.313890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:43956)
0808 13:00:04.314074 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.314149 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 13:00:04.314534 13468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:35005: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
I20260808 13:00:04.382789 13468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } attempt: 2
I20260808 13:00:04.383965 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.383489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:43962)
0808 13:00:04.383696 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.383801 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 13:00:04.384981 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.384637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:43968)
0808 13:00:04.384784 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.384860 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 13:00:04.385185 13468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:35005: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
I20260808 13:00:04.405395 13349 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } attempt: 5
Aug 08 13:00:04 dist-test-slave-tk9z krb5kdc[13216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194003, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for kudu/127.24.167.61@KRBTEST.COM
I20260808 13:00:04.419375 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.406225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:46515 (local address 127.0.0.1:37298)
0808 13:00:04.406532 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.406555 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:04.406703 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:04.407126 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:04.407130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:04.407576 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:04.407769 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.407778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.408799 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.408806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.409719 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.409731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.410576 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.410582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.410789 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.411459 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:04.411478 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:04.413639 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:04.416813 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.416821 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.416825 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.417240 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.417646 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.417651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.417655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.417849 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.418605 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:04.418609 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:04.418759 (+ 150us) client_negotiation.cc:769] Sending connection context
0808 13:00:04.418884 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 13:00:04.419108 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260808 13:00:04.419658 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.406374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.24.167.61:46515)
0808 13:00:04.406614 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:04.406620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:04.406643 (+ 23us) server_negotiation.cc:407] Connection header received
0808 13:00:04.406902 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:04.406906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:04.406972 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:04.407086 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:04.407942 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.408653 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.409874 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.410364 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.410565 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.413888 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:04.413893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:04.413897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:04.413932 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:04.416655 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.417394 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.417399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.417402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.417526 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.418019 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.418025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.418029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.418465 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:04.418622 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:04.419069 (+ 447us) server_negotiation.cc:299] Negotiation successful
0808 13:00:04.419335 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260808 13:00:04.420778 13349 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } has no permanent_uuid. Determining permanent_uuid...
I20260808 13:00:04.421901 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.421497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:43984)
0808 13:00:04.421669 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.421758 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 13:00:04.422852 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.422519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:43998)
0808 13:00:04.422656 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.422746 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 13:00:04.423081 13349 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:35005: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
I20260808 13:00:04.474292 13349 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } attempt: 1
I20260808 13:00:04.475768 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.475052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:44000)
0808 13:00:04.475341 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.475508 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 13:00:04.476415 13468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } attempt: 3
I20260808 13:00:04.477049 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.476632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:44016)
0808 13:00:04.476848 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.476923 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260808 13:00:04.477281 13349 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:35005: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
I20260808 13:00:04.477780 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.477096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:44018)
0808 13:00:04.477450 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.477576 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260808 13:00:04.479048 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.478485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:44032)
0808 13:00:04.478761 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.478875 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260808 13:00:04.479352 13468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:35005: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
I20260808 13:00:04.545569 13349 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } attempt: 2
I20260808 13:00:04.546993 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.546318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:44034)
0808 13:00:04.546638 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.546769 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260808 13:00:04.548157 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.547730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:44042)
0808 13:00:04.547904 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.547993 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 13:00:04.548393 13349 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:35005: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
I20260808 13:00:04.622614 13468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } attempt: 4
W20260808 13:00:04.622941 13480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:04.623299 13480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:04.623479 13480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 13:00:04.623925 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.623338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:44046)
0808 13:00:04.623625 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.623739 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 13:00:04.624946 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.624589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:44054)
0808 13:00:04.624741 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.624822 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 13:00:04.625173 13468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:35005: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
W20260808 13:00:04.631511 13480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:00:04.631608 13480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:04.631666 13480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:04.631717 13480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:00:04.631762 13480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:00:04.642140 13480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:41633,127.24.167.61:46515,127.24.167.60:35005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.60.keytab
--principal=kudu/127.24.167.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.60:35005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.167.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:04.643622 13480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:04.644955 13480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 13:00:04.654592 13349 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } attempt: 3
I20260808 13:00:04.655733 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.655278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:44058)
0808 13:00:04.655493 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.655580 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 13:00:04.656997 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.656468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:44068)
0808 13:00:04.656762 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.656855 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260808 13:00:04.657199 13488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:04.657320 13487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:04.657378 13349 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:35005: Network error: Client connection negotiation failed: client connection to 127.24.167.60:35005: connect: Connection refused (error 111)
W20260808 13:00:04.658034 13492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:04.658854 13480 server_base.cc:1034] running on GCE node
I20260808 13:00:04.659591 13480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:04.661008 13480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:04.662238 13480 hybrid_clock.cc:648] HybridClock initialized: now 1786194004662178 us; error 46 us; skew 500 ppm
I20260808 13:00:04.662639 13480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:04 dist-test-slave-tk9z krb5kdc[13216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194004, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:04.665866 13480 init.cc:377] Logged in from keytab as kudu/127.24.167.60@KRBTEST.COM (short username kudu)
I20260808 13:00:04.667367 13480 webserver.cc:460] Webserver started at http://127.24.167.60:36485/ using document root <none> and password file <none>
I20260808 13:00:04.668097 13480 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:04.668256 13480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:04.668588 13480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:00:04.671804 13480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/data/instance:
uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2"
format_stamp: "Formatted at 2026-08-08 13:00:04 on dist-test-slave-tk9z"
server_key: "467304171423831953086fdce3b8365e"
server_key_iv: "345d5b18813028d71e40971c938463dd"
server_key_version: "encryptionkey@0"
I20260808 13:00:04.672659 13480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/wal/instance:
uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2"
format_stamp: "Formatted at 2026-08-08 13:00:04 on dist-test-slave-tk9z"
server_key: "467304171423831953086fdce3b8365e"
server_key_iv: "345d5b18813028d71e40971c938463dd"
server_key_version: "encryptionkey@0"
I20260808 13:00:04.678103 13480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 13:00:04.681681 13499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:04.683169 13480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:00:04.683359 13480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/wal
uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2"
format_stamp: "Formatted at 2026-08-08 13:00:04 on dist-test-slave-tk9z"
server_key: "467304171423831953086fdce3b8365e"
server_key_iv: "345d5b18813028d71e40971c938463dd"
server_key_version: "encryptionkey@0"
I20260808 13:00:04.683533 13480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:04.729017 13480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:04.733340 13480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:04.733604 13480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:04.752965 13480 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.60:35005
I20260808 13:00:04.753001 13597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.60:35005 every 8 connection(s)
I20260808 13:00:04.754509 13480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/data/info.pb
I20260808 13:00:04.758402 13598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:00:04.760224 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 13480
I20260808 13:00:04.760417 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786193784050718-25244-0/minicluster-data/master-2/wal/instance
I20260808 13:00:04.760926 25244 external_mini_cluster.cc:1428] Setting key 6c592e3d3e09a933792245f6c9921c74
I20260808 13:00:04.768401 13598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 13:00:04 dist-test-slave-tk9z krb5kdc[13216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.60@KRBTEST.COM
Aug 08 13:00:04 dist-test-slave-tk9z krb5kdc[13216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194004, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.60@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:00:04.786983 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.763027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.24.167.60:35005)
0808 13:00:04.763946 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:04.763967 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:04.764023 (+ 56us) server_negotiation.cc:407] Connection header received
0808 13:00:04.765725 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:04.765756 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:04.766299 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:04.766764 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:04.768154 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.770187 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.772251 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.773283 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.773504 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.778523 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:04.778533 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:04.778546 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:04.778587 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:04.781776 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.782866 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.782874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.782881 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.783054 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.783623 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.783628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.783630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.784240 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:04.784589 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:04.785126 (+ 537us) server_negotiation.cc:299] Negotiation successful
0808 13:00:04.785654 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":403,"threads_started":1}
I20260808 13:00:04.788300 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.771463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41633 (local address 127.0.0.1:52384)
0808 13:00:04.772330 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.772397 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:04.772574 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:04.773170 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:04.773175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:04.773804 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:04.774164 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.774181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.775853 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.775857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.777138 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.777156 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.778246 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.778255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.778527 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.779327 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:04.779347 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:04.781915 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:04.785274 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.785286 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.785294 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.785901 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.786296 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.786301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.786304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.786543 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.787099 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:04.787106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:04.787349 (+ 243us) client_negotiation.cc:769] Sending connection context
0808 13:00:04.787546 (+ 197us) client_negotiation.cc:240] Negotiation successful
0808 13:00:04.787860 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":322,"threads_started":1}
I20260808 13:00:04.788431 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.771497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.24.167.62:41633)
0808 13:00:04.772796 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:04.772802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:04.772822 (+ 20us) server_negotiation.cc:407] Connection header received
0808 13:00:04.772888 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:04.772894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:04.772993 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:04.773109 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:04.774372 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.775687 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.777412 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.777954 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.778124 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.782156 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:04.782163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:04.782168 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:04.782214 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:04.785126 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.786028 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.786034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.786038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.786175 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.786647 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.786652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.786660 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.786968 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:04.787142 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:04.787698 (+ 556us) server_negotiation.cc:299] Negotiation successful
0808 13:00:04.787937 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1157,"thread_start_us":1039,"threads_started":1}
I20260808 13:00:04.790005 13598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 13:00:04 dist-test-slave-tk9z krb5kdc[13216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194004, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.60@KRBTEST.COM for kudu/127.24.167.61@KRBTEST.COM
Aug 08 13:00:04 dist-test-slave-tk9z krb5kdc[13216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:00:04.805316 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.791203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.24.167.61:46515)
0808 13:00:04.791418 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:04.791422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:04.791552 (+ 130us) server_negotiation.cc:407] Connection header received
0808 13:00:04.791699 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:04.791706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:04.791791 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:04.791900 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:04.792564 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.793590 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.795027 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.795688 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.795789 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.799124 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:04.799130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:04.799135 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:04.799178 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:04.802626 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.803372 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.803378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.803382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.803547 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.803960 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.803966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.803972 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.804325 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:04.804494 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:04.804892 (+ 398us) server_negotiation.cc:299] Negotiation successful
0808 13:00:04.805054 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 13:00:04.805444 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.790772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:46515 (local address 127.0.0.1:37310)
0808 13:00:04.791463 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.791482 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:04.791581 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:04.791927 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:04.791930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:04.792258 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:04.792407 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.792414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.793732 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.793738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.794888 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.794900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.795832 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.795837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.796049 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.796716 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:04.796733 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:04.798915 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:04.802754 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.802758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.802760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.803248 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.803694 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.803698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.803700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.803839 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.804461 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:04.804465 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:04.804619 (+ 154us) client_negotiation.cc:769] Sending connection context
0808 13:00:04.804798 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 13:00:04.805073 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532}
I20260808 13:00:04.806788 13598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } has no permanent_uuid. Determining permanent_uuid...
I20260808 13:00:04.807549 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.790912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.24.167.62:41633)
0808 13:00:04.791103 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:04.791108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:04.791166 (+ 58us) server_negotiation.cc:407] Connection header received
0808 13:00:04.791352 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:04.791358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:04.791452 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:04.791561 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:04.793142 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.794084 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.795660 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.796305 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.796409 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.801167 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:04.801175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:04.801180 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:04.801231 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:04.804604 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.805494 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.805500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.805503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.805653 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.806059 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.806064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.806067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.806422 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:04.806721 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:04.807018 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 13:00:04.807193 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Aug 08 13:00:04 dist-test-slave-tk9z krb5kdc[13216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194004, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.60@KRBTEST.COM for kudu/127.24.167.60@KRBTEST.COM
Aug 08 13:00:04 dist-test-slave-tk9z krb5kdc[13216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.61@KRBTEST.COM
I20260808 13:00:04.821096 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.807906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.24.167.60:35005)
0808 13:00:04.808081 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:04.808087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:04.808103 (+ 16us) server_negotiation.cc:407] Connection header received
0808 13:00:04.808153 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:04.808157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:04.808228 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:04.808300 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:04.809135 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.810213 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.811508 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.811922 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.812012 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.815586 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:04.815590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:04.815594 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:04.815621 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:04.818243 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.819064 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.819071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.819075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.819224 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.819628 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.819633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.819637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.820011 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:04.820152 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:04.820492 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 13:00:04.820718 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 13:00:04.821301 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.807709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:44082)
0808 13:00:04.807900 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.807917 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:04.808022 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:04.808361 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:04.808366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:04.808847 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:04.808995 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.809003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.810357 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.810361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.811354 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.811372 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.812060 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.812066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.812277 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.812853 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:04.812869 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:04.815399 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:04.818423 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.818428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.818433 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.818931 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.819328 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.819331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.819334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.819498 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.820115 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:04.820120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:04.820250 (+ 130us) client_negotiation.cc:769] Sending connection context
0808 13:00:04.820402 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 13:00:04.820865 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":148}
I20260808 13:00:04.824415 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.809546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.24.167.61:46515)
0808 13:00:04.809711 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:04.809715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:04.809729 (+ 14us) server_negotiation.cc:407] Connection header received
0808 13:00:04.809767 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:04.809770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:04.809817 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:04.809890 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:04.811280 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.812137 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.813420 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.814140 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.814235 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.818581 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:04.818586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:04.818589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:04.818636 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:04.821570 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.822278 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.822284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.822289 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.822496 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.822938 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.822943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.822946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.823293 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:04.823600 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:04.823814 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 13:00:04.824070 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 13:00:04.827702 13598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2: Bootstrap starting.
I20260808 13:00:04.829581 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:00:04.831856 13598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2: Neither blocks nor log segments found. Creating new log.
I20260808 13:00:04.833323 13598 log.cc:824] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:04.834580 13349 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } attempt: 4
I20260808 13:00:04.836928 13598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2: No bootstrap required, opened a new log
Aug 08 13:00:04 dist-test-slave-tk9z krb5kdc[13216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194003, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for kudu/127.24.167.60@KRBTEST.COM
I20260808 13:00:04.859546 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.835236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:44098)
0808 13:00:04.835416 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.835437 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:04.835559 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:04.835881 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:04.835888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:04.836379 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:04.836635 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.836649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.838598 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.838607 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.840671 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.840693 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.842476 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.842486 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.842763 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.843629 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:04.843659 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:04.847322 (+ 3663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:04.856075 (+ 8753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.856084 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.856092 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.856823 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.857461 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.857469 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.857473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.857659 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.858482 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:04.858487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:04.858666 (+ 179us) client_negotiation.cc:769] Sending connection context
0808 13:00:04.858801 (+ 135us) client_negotiation.cc:240] Negotiation successful
0808 13:00:04.859138 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 13:00:04.860569 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.835371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44098 (local address 127.24.167.60:35005)
0808 13:00:04.835578 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:04.835584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:04.835605 (+ 21us) server_negotiation.cc:407] Connection header received
0808 13:00:04.835667 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:04.835672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:04.835739 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:04.835842 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:04.836815 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.838260 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.840914 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.841972 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.842128 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.847591 (+ 5463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:04.847601 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:04.847609 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:04.847661 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:04.855834 (+ 8173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.857065 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.857074 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.857081 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.857302 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.857811 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.857817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.857821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.858259 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:04.859840 (+ 1581us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:04.860006 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 13:00:04.860222 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 13:00:04.864017 13598 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } }
I20260808 13:00:04.864535 13598 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:00:04.864665 13598 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a7af84b1b141d3bd1f0da5bfa887a2, State: Initialized, Role: FOLLOWER
I20260808 13:00:04.866977 13349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c: Bootstrap starting.
I20260808 13:00:04.871002 13598 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } }
I20260808 13:00:04.871634 13349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c: Neither blocks nor log segments found. Creating new log.
I20260808 13:00:04.873133 13349 log.cc:824] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:04.876957 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.837259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52400 (local address 127.24.167.62:41633)
0808 13:00:04.837430 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:04.837436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:04.846060 (+ 8624us) server_negotiation.cc:407] Connection header received
0808 13:00:04.846328 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:04.846336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:04.846438 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:04.846530 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:04.849758 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.851571 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.855004 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.855661 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.855777 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.865236 (+ 9459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:04.865247 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:04.865255 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:04.865302 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:04.869210 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.871773 (+ 2563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.871778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.871781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.871947 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.874300 (+ 2353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.874310 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.874316 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.874878 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:04.875226 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:04.875986 (+ 760us) server_negotiation.cc:299] Negotiation successful
0808 13:00:04.876379 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 13:00:04.883857 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.836841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44108 (local address 127.24.167.60:35005)
0808 13:00:04.857805 (+ 20964us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:04.857813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:04.857840 (+ 27us) server_negotiation.cc:407] Connection header received
0808 13:00:04.857901 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:04.857909 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:04.858018 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:04.858149 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:04.861256 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.862895 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.865836 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.866826 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.866966 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.871233 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:04.871241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:04.871247 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:04.871288 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:04.875924 (+ 4636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.877249 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.877255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.877258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.877509 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.878140 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.878149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.878155 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.880330 (+ 2175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:04.880679 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:04.882348 (+ 1669us) server_negotiation.cc:299] Negotiation successful
0808 13:00:04.882702 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20768,"thread_start_us":16269,"threads_started":1}
I20260808 13:00:04.884608 13625 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } } }
I20260808 13:00:04.885094 13625 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:00:04.886972 13598 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:00:04.894169 13349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c: No bootstrap required, opened a new log
I20260808 13:00:04.896723 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.843958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.24.167.61:46515)
0808 13:00:04.844116 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:04.844120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:04.871007 (+ 26887us) server_negotiation.cc:407] Connection header received
0808 13:00:04.871368 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:04.871376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:04.871476 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:04.871562 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:04.878784 (+ 7222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.880308 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.883138 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.884064 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.884159 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.888905 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:04.888916 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:04.888923 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:04.888983 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:04.892832 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.894046 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.894052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.894055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.894230 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.894811 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.894817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.894820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.895250 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:04.895485 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:04.896097 (+ 612us) server_negotiation.cc:299] Negotiation successful
0808 13:00:04.896359 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260808 13:00:04.902063 13651 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 13:00:04.902223 13651 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 13:00:04.905511 13349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } }
I20260808 13:00:04.905957 13349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:00:04.906088 13349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f4b92de695b4754b6c39436569d0f5c, State: Initialized, Role: FOLLOWER
I20260808 13:00:04.906807 13349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } }
I20260808 13:00:04.896435 13480 master_runner.cc:186] Error getting master registration for 127.24.167.62:41633: OK, instance_id { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" instance_seqno: 1786194003686116 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 13:00:04.908696 13480 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 13:00:04.914216 13654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } } }
I20260808 13:00:04.914727 13654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:00:04.916071 13349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:00:04.924386 13468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } attempt: 5
W20260808 13:00:04.924443 13271 tablet.cc:2369] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 13:00:04.929158 13672 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 13:00:04.929399 13672 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 08 13:00:04 dist-test-slave-tk9z krb5kdc[13216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194004, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.61@KRBTEST.COM for kudu/127.24.167.60@KRBTEST.COM
I20260808 13:00:04.942263 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.925042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:35005 (local address 127.0.0.1:44118)
0808 13:00:04.925400 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:04.925417 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:04.925506 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:04.926273 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:04.926277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:04.926874 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:04.927089 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.927106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.928553 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.928559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.929826 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:04.929840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.930933 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.930940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:04.931198 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.931910 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:04.931936 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:04.934943 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:04.939240 (+ 4297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.939258 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.939263 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.939812 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.940295 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:04.940301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:04.940305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:04.940475 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:04.941204 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:04.941209 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:04.941370 (+ 161us) client_negotiation.cc:769] Sending connection context
0808 13:00:04.941532 (+ 162us) client_negotiation.cc:240] Negotiation successful
0808 13:00:04.941838 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260808 13:00:04.943368 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:04.925735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.24.167.60:35005)
0808 13:00:04.925949 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:04.925955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:04.925972 (+ 17us) server_negotiation.cc:407] Connection header received
0808 13:00:04.926029 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:04.926037 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:04.926128 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:04.926238 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:04.927271 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.928373 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.929974 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:04.930551 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:04.930689 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:04.935161 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:04.935168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:04.935173 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:04.935213 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:04.939068 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.939982 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.939988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.939992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.940156 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:04.940615 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:04.940621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:04.940624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:04.940996 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:04.941141 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:04.942739 (+ 1598us) server_negotiation.cc:299] Negotiation successful
0808 13:00:04.943027 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 13:00:04.926932 13238 master_runner.cc:186] Error getting master registration for 127.24.167.61:46515: OK, instance_id { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" instance_seqno: 1786194004197018 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 13:00:04.943780 13238 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 13:00:04.949978 13468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407: Bootstrap starting.
I20260808 13:00:04.955556 13468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407: Neither blocks nor log segments found. Creating new log.
I20260808 13:00:04.956794 13468 log.cc:824] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:04.959684 13468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407: No bootstrap required, opened a new log
I20260808 13:00:04.968511 13468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } }
I20260808 13:00:04.968873 13468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:00:04.969033 13468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2adb5cdb9fa6462fb8f2ec00c70c7407, State: Initialized, Role: FOLLOWER
I20260808 13:00:04.969704 13468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } }
I20260808 13:00:04.970949 13677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } } }
I20260808 13:00:04.971434 13677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:00:04.972199 13468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 13:00:04.985756 13513 tablet.cc:2369] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:00:05.001339 13360 master_runner.cc:263] No leader master found from master 2adb5cdb9fa6462fb8f2ec00c70c7407
I20260808 13:00:05.001510 13360 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 13:00:05.005236 13694 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 13:00:05.005537 13694 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 13:00:05.166266 13625 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 13:00:05.166702 13625 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } }
I20260808 13:00:05.168536 13625 leader_election.cc:288] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f4b92de695b4754b6c39436569d0f5c (127.24.167.62:41633), 2adb5cdb9fa6462fb8f2ec00c70c7407 (127.24.167.61:46515)
I20260808 13:00:05.170003 13306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f4b92de695b4754b6c39436569d0f5c" is_pre_election: true
I20260808 13:00:05.171330 13306 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2a7af84b1b141d3bd1f0da5bfa887a2 in term 0.
I20260808 13:00:05.172762 13501 leader_election.cc:302] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f4b92de695b4754b6c39436569d0f5c, d2a7af84b1b141d3bd1f0da5bfa887a2; no voters:
I20260808 13:00:05.173693 13434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" is_pre_election: true
I20260808 13:00:05.174278 13434 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2a7af84b1b141d3bd1f0da5bfa887a2 in term 0.
I20260808 13:00:05.175501 13625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 13:00:05.175658 13625 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 13:00:05.175758 13625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:00:05.179318 13625 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } }
I20260808 13:00:05.180042 13625 leader_election.cc:288] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [CANDIDATE]: Term 1 election: Requested vote from peers 5f4b92de695b4754b6c39436569d0f5c (127.24.167.62:41633), 2adb5cdb9fa6462fb8f2ec00c70c7407 (127.24.167.61:46515)
I20260808 13:00:05.180964 13306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f4b92de695b4754b6c39436569d0f5c"
I20260808 13:00:05.181211 13306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:00:05.185364 13306 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2a7af84b1b141d3bd1f0da5bfa887a2 in term 1.
I20260808 13:00:05.186213 13434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407"
I20260808 13:00:05.186578 13434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:00:05.190253 13434 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2a7af84b1b141d3bd1f0da5bfa887a2 in term 1.
W20260808 13:00:05.190653 13394 tablet.cc:2369] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:00:05.192203 13501 leader_election.cc:302] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2adb5cdb9fa6462fb8f2ec00c70c7407, d2a7af84b1b141d3bd1f0da5bfa887a2; no voters:
I20260808 13:00:05.193289 13625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:00:05.202708 13625 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [term 1 LEADER]: Becoming Leader. State: Replica: d2a7af84b1b141d3bd1f0da5bfa887a2, State: Running, Role: LEADER
I20260808 13:00:05.203395 13625 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } }
I20260808 13:00:05.213665 13696 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2a7af84b1b141d3bd1f0da5bfa887a2. Latest consensus state: current_term: 1 leader_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } } }
I20260808 13:00:05.214231 13696 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [sys.catalog]: This master's current role is: LEADER
I20260808 13:00:05.239013 13698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:00:05.241556 13698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:00:05.264482 13434 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407 [term 1 FOLLOWER]: Refusing update from remote peer d2a7af84b1b141d3bd1f0da5bfa887a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:00:05.265805 13306 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c [term 1 FOLLOWER]: Refusing update from remote peer d2a7af84b1b141d3bd1f0da5bfa887a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:00:05.267359 13625 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:00:05.268177 13625 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:00:05.315689 13654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2a7af84b1b141d3bd1f0da5bfa887a2. Latest consensus state: current_term: 1 leader_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } } }
I20260808 13:00:05.316090 13654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:00:05.320562 13699 mvcc.cc:205] Tried to move back new op lower bound from 7316250645548826624 to 7316250645319020544. Current Snapshot: MvccSnapshot[applied={T|T < 7316250645548826624}]
I20260808 13:00:05.326821 13625 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } } }
I20260808 13:00:05.327189 13625 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [sys.catalog]: This master's current role is: LEADER
I20260808 13:00:05.329632 13698 catalog_manager.cc:1347] Generated new cluster ID: 82b5e08efefb4d189da59a58c1cae02e
I20260808 13:00:05.329753 13698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:00:05.333642 13703 mvcc.cc:205] Tried to move back new op lower bound from 7316250645548826624 to 7316250645319020544. Current Snapshot: MvccSnapshot[applied={T|T < 7316250645548826624}]
I20260808 13:00:05.335620 13677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2a7af84b1b141d3bd1f0da5bfa887a2. Latest consensus state: current_term: 1 leader_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } } }
I20260808 13:00:05.335981 13677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:00:05.341950 13696 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } } }
I20260808 13:00:05.342252 13696 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 [sys.catalog]: This master's current role is: LEADER
I20260808 13:00:05.342738 13701 mvcc.cc:205] Tried to move back new op lower bound from 7316250645548826624 to 7316250645319020544. Current Snapshot: MvccSnapshot[applied={T|T < 7316250645548826624}]
I20260808 13:00:05.352128 13654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } } }
I20260808 13:00:05.352473 13654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:00:05.352445 13677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f4b92de695b4754b6c39436569d0f5c" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41633 } } peers { permanent_uuid: "2adb5cdb9fa6462fb8f2ec00c70c7407" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 46515 } } peers { permanent_uuid: "d2a7af84b1b141d3bd1f0da5bfa887a2" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 35005 } } }
I20260808 13:00:05.352725 13677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:00:05.366159 13698 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:00:05.368194 13698 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:00:05.394526 13698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2: Generated new TSK 0
I20260808 13:00:05.395694 13698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:00:05.931723 13672 catalog_manager.cc:1259] Loaded cluster ID: 82b5e08efefb4d189da59a58c1cae02e
I20260808 13:00:05.931836 13672 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c: loading cluster ID for follower catalog manager: success
I20260808 13:00:05.934681 13672 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c: acquiring CA information for follower catalog manager: success
I20260808 13:00:05.936621 13672 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5f4b92de695b4754b6c39436569d0f5c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 13:00:06.007740 13694 catalog_manager.cc:1259] Loaded cluster ID: 82b5e08efefb4d189da59a58c1cae02e
I20260808 13:00:06.007851 13694 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407: loading cluster ID for follower catalog manager: success
I20260808 13:00:06.010725 13694 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407: acquiring CA information for follower catalog manager: success
I20260808 13:00:06.012645 13694 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2adb5cdb9fa6462fb8f2ec00c70c7407: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 13:00:06.037626 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 13238
I20260808 13:00:06.038879 13340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:06.152164 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 13238
I20260808 13:00:06.184980 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 13360
I20260808 13:00:06.186169 13462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:06.249691 13724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:06.248854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41633 (local address 127.0.0.1:52408)
0808 13:00:06.249380 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:06.249510 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":299,"threads_started":1}
W20260808 13:00:06.250049 13501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41633: connect: Connection refused (error 111)
I20260808 13:00:06.251657 13724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:06.251275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41633 (local address 127.0.0.1:52412)
0808 13:00:06.251462 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:06.251540 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 13:00:06.251861 13501 consensus_peers.cc:487] T 00000000000000000000000000000000 P d2a7af84b1b141d3bd1f0da5bfa887a2 -> Peer 5f4b92de695b4754b6c39436569d0f5c (127.24.167.62:41633): Couldn't send request to peer 5f4b92de695b4754b6c39436569d0f5c. Status: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41633: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 13:00:06.295820 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 13360
I20260808 13:00:06.324662 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 13480
I20260808 13:00:06.325888 13589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:06.435369 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 13480
2026-08-08T13:00:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4832 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:00:06 dist-test-slave-tk9z krb5kdc[13730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:00:06 dist-test-slave-tk9z krb5kdc[13730](info): set up 2 sockets
Aug 08 13:00:06 dist-test-slave-tk9z krb5kdc[13730](info): commencing operation
krb5kdc: starting...
W20260808 13:00:07.859875 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.373s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:00:08 dist-test-slave-tk9z krb5kdc[13730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:00:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:00:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 13:00:08.156224 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:40621
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:38511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:40621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:00:08.507151 13752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:08.507469 13752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:08.507649 13752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:08.514986 13752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:00:08.515108 13752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:08.515197 13752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:08.515307 13752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:00:08.515391 13752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:00:08.525542 13752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:38511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:40621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:40621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:08.527027 13752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:08.528393 13752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:08.540900 13758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:08.541019 13757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:08.540987 13760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:08.541692 13752 server_base.cc:1034] running on GCE node
I20260808 13:00:08.542644 13752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:08.544329 13752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:08.545580 13752 hybrid_clock.cc:648] HybridClock initialized: now 1786194008545540 us; error 51 us; skew 500 ppm
I20260808 13:00:08.545972 13752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:08 dist-test-slave-tk9z krb5kdc[13730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194008, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:08.549206 13752 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:00:08.550807 13752 webserver.cc:460] Webserver started at http://127.24.167.62:34363/ using document root <none> and password file <none>
I20260808 13:00:08.551537 13752 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:08.551697 13752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:08.552027 13752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:00:08.555323 13752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "ee56721dd13e4c2c9bf4e7d697f6a2a1"
format_stamp: "Formatted at 2026-08-08 13:00:08 on dist-test-slave-tk9z"
server_key: "c415d5b92ac58ff442ee78ef5ca2be17"
server_key_iv: "42d73ca2845ddf4a780986796ac33db0"
server_key_version: "encryptionkey@0"
I20260808 13:00:08.556136 13752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "ee56721dd13e4c2c9bf4e7d697f6a2a1"
format_stamp: "Formatted at 2026-08-08 13:00:08 on dist-test-slave-tk9z"
server_key: "c415d5b92ac58ff442ee78ef5ca2be17"
server_key_iv: "42d73ca2845ddf4a780986796ac33db0"
server_key_version: "encryptionkey@0"
I20260808 13:00:08.561863 13752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 13:00:08.566111 13769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:08.567557 13752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:00:08.567775 13752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "ee56721dd13e4c2c9bf4e7d697f6a2a1"
format_stamp: "Formatted at 2026-08-08 13:00:08 on dist-test-slave-tk9z"
server_key: "c415d5b92ac58ff442ee78ef5ca2be17"
server_key_iv: "42d73ca2845ddf4a780986796ac33db0"
server_key_version: "encryptionkey@0"
I20260808 13:00:08.568013 13752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:08.596829 13752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:08.601047 13752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:08.601385 13752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:08.621595 13752 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:40621
I20260808 13:00:08.621614 13857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:40621 every 8 connection(s)
I20260808 13:00:08.623119 13752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:00:08.624948 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 13752
I20260808 13:00:08.625204 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:00:08.625938 25244 external_mini_cluster.cc:1428] Setting key ee3fff9300efa5de68c452c57688943d
I20260808 13:00:08.627627 13859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 13:00:08 dist-test-slave-tk9z krb5kdc[13730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:00:08.642568 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1: Bootstrap starting.
I20260808 13:00:08.647225 13859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1: Neither blocks nor log segments found. Creating new log.
I20260808 13:00:08.648784 13859 log.cc:824] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:08.652954 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1: No bootstrap required, opened a new log
I20260808 13:00:08.654085 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:08.628044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.24.167.62:40621)
0808 13:00:08.628912 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:08.628939 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:08.628994 (+ 55us) server_negotiation.cc:407] Connection header received
0808 13:00:08.631213 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:08.631244 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:08.631801 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:08.632297 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:08.633803 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:08.635973 (+ 2170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:08.637389 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:08.638500 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:08.638723 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:08.643683 (+ 4960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:08.643697 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:08.643722 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:08.643786 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:08.648310 (+ 4524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:08.649394 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:08.649408 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:08.649422 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:08.649722 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:08.650319 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:08.650322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:08.650325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:08.651195 (+ 870us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:08.651507 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:08.652078 (+ 571us) server_negotiation.cc:299] Negotiation successful
0808 13:00:08.652558 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":349,"threads_started":1}
I20260808 13:00:08.658557 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:00:08.661727 13859 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee56721dd13e4c2c9bf4e7d697f6a2a1" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 40621 } }
I20260808 13:00:08.662221 13859 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:00:08.662343 13859 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee56721dd13e4c2c9bf4e7d697f6a2a1, State: Initialized, Role: FOLLOWER
I20260808 13:00:08.663066 13859 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee56721dd13e4c2c9bf4e7d697f6a2a1" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 40621 } }
I20260808 13:00:08.663322 13859 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:00:08.663465 13859 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:00:08.663655 13859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:00:08.665942 13859 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee56721dd13e4c2c9bf4e7d697f6a2a1" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 40621 } }
I20260808 13:00:08.666597 13859 leader_election.cc:302] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee56721dd13e4c2c9bf4e7d697f6a2a1; no voters:
I20260808 13:00:08.667235 13859 leader_election.cc:288] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:00:08.667429 13868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:00:08.668215 13868 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [term 1 LEADER]: Becoming Leader. State: Replica: ee56721dd13e4c2c9bf4e7d697f6a2a1, State: Running, Role: LEADER
I20260808 13:00:08.668814 13868 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee56721dd13e4c2c9bf4e7d697f6a2a1" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 40621 } }
I20260808 13:00:08.669481 13859 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:00:08.671723 13870 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee56721dd13e4c2c9bf4e7d697f6a2a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee56721dd13e4c2c9bf4e7d697f6a2a1" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 40621 } } }
I20260808 13:00:08.672413 13870 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [sys.catalog]: This master's current role is: LEADER
I20260808 13:00:08.672297 13871 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee56721dd13e4c2c9bf4e7d697f6a2a1. Latest consensus state: current_term: 1 leader_uuid: "ee56721dd13e4c2c9bf4e7d697f6a2a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee56721dd13e4c2c9bf4e7d697f6a2a1" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 40621 } } }
I20260808 13:00:08.672587 13871 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1 [sys.catalog]: This master's current role is: LEADER
I20260808 13:00:08.673434 13878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:00:08.678256 13878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:00:08.689008 13878 catalog_manager.cc:1347] Generated new cluster ID: 507512925a214c5da764f975d9b11e6d
I20260808 13:00:08.689177 13878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:00:08.700039 13878 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:00:08.701529 13878 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:00:08.714154 13878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee56721dd13e4c2c9bf4e7d697f6a2a1: Generated new TSK 0
I20260808 13:00:08.715384 13878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:00:08.772176 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 13752
I20260808 13:00:08.773587 13852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:08.875082 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 13752
2026-08-08T13:00:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2435 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:00:08 dist-test-slave-tk9z krb5kdc[13898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:00:08 dist-test-slave-tk9z krb5kdc[13898](info): set up 2 sockets
Aug 08 13:00:08 dist-test-slave-tk9z krb5kdc[13898](info): commencing operation
krb5kdc: starting...
W20260808 13:00:10.223577 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.301s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:00:10 dist-test-slave-tk9z krb5kdc[13898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:00:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:00:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 13:00:10.511540 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:41391
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:41391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:00:10.871495 13920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:10.871797 13920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:10.871891 13920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:10.879360 13920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:00:10.879487 13920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:10.879535 13920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:10.879586 13920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:00:10.879631 13920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:00:10.889616 13920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:41391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:10.891017 13920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:10.892375 13920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:10.905200 13926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:10.905200 13932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:10.905560 13928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:10.907635 13920 server_base.cc:1034] running on GCE node
I20260808 13:00:10.908644 13920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:10.910280 13920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:10.911581 13920 hybrid_clock.cc:648] HybridClock initialized: now 1786194010911518 us; error 51 us; skew 500 ppm
I20260808 13:00:10.912120 13920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:10 dist-test-slave-tk9z krb5kdc[13898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194010, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:10.916487 13920 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:00:10.919039 13920 webserver.cc:460] Webserver started at http://127.24.167.62:36235/ using document root <none> and password file <none>
I20260808 13:00:10.920190 13920 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:10.920378 13920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:10.920708 13920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:00:10.924304 13920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "0520edffaf2a4ee1ac7a31a6011424ae"
format_stamp: "Formatted at 2026-08-08 13:00:10 on dist-test-slave-tk9z"
server_key: "c16da6c002060beaa75c4da425a758fc"
server_key_iv: "cd0408edbd7004997948db7f8ee18468"
server_key_version: "encryptionkey@0"
I20260808 13:00:10.925132 13920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "0520edffaf2a4ee1ac7a31a6011424ae"
format_stamp: "Formatted at 2026-08-08 13:00:10 on dist-test-slave-tk9z"
server_key: "c16da6c002060beaa75c4da425a758fc"
server_key_iv: "cd0408edbd7004997948db7f8ee18468"
server_key_version: "encryptionkey@0"
I20260808 13:00:10.931041 13920 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260808 13:00:10.934782 13940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:10.936174 13920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 13:00:10.936371 13920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "0520edffaf2a4ee1ac7a31a6011424ae"
format_stamp: "Formatted at 2026-08-08 13:00:10 on dist-test-slave-tk9z"
server_key: "c16da6c002060beaa75c4da425a758fc"
server_key_iv: "cd0408edbd7004997948db7f8ee18468"
server_key_version: "encryptionkey@0"
I20260808 13:00:10.936580 13920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:10.975230 13920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:10.979926 13920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:10.980262 13920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:11.000753 13920 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:41391
I20260808 13:00:11.000794 14033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:41391 every 8 connection(s)
I20260808 13:00:11.002295 13920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:00:11.006546 14035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:00:11.009214 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 13920
I20260808 13:00:11.009471 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:00:11.010339 25244 external_mini_cluster.cc:1428] Setting key eb478cea282c21c08d76678e0f8d72d6
I20260808 13:00:11.020608 14035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: Bootstrap starting.
I20260808 13:00:11.024820 14035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: Neither blocks nor log segments found. Creating new log.
I20260808 13:00:11.025893 14035 log.cc:824] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: Log is configured to *not* fsync() on all Append() calls
Aug 08 13:00:11 dist-test-slave-tk9z krb5kdc[13898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:00:11.029142 14035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: No bootstrap required, opened a new log
I20260808 13:00:11.035431 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:11.012737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60434 (local address 127.24.167.62:41391)
0808 13:00:11.013657 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:11.013669 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:11.013715 (+ 46us) server_negotiation.cc:407] Connection header received
0808 13:00:11.015500 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:11.015526 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:11.015908 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:11.016230 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:11.018057 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:11.019474 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:11.021442 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:11.022684 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:11.022885 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:11.027873 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:11.027880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:11.027896 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:11.027937 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:11.031228 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:11.031941 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:11.031955 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:11.031961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:11.032125 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:11.032534 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:11.032539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:11.032542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:11.033111 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:11.033492 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:11.033950 (+ 458us) server_negotiation.cc:299] Negotiation successful
0808 13:00:11.034411 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":442,"threads_started":1}
I20260808 13:00:11.037516 14035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } }
I20260808 13:00:11.037930 14035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:00:11.038053 14035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0520edffaf2a4ee1ac7a31a6011424ae, State: Initialized, Role: FOLLOWER
I20260808 13:00:11.038704 14035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } }
I20260808 13:00:11.038975 14035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:00:11.039129 14035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:00:11.039297 14035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:00:11.041944 14035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } }
I20260808 13:00:11.042562 14035 leader_election.cc:302] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0520edffaf2a4ee1ac7a31a6011424ae; no voters:
I20260808 13:00:11.043210 14035 leader_election.cc:288] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:00:11.043345 14044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:00:11.044165 14044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 1 LEADER]: Becoming Leader. State: Replica: 0520edffaf2a4ee1ac7a31a6011424ae, State: Running, Role: LEADER
I20260808 13:00:11.044744 14044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } }
I20260808 13:00:11.045567 14035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:00:11.047912 14047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0520edffaf2a4ee1ac7a31a6011424ae. Latest consensus state: current_term: 1 leader_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } } }
I20260808 13:00:11.051131 14047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [sys.catalog]: This master's current role is: LEADER
I20260808 13:00:11.047876 14046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } } }
I20260808 13:00:11.051695 14046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [sys.catalog]: This master's current role is: LEADER
I20260808 13:00:11.052088 14053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:00:11.057672 14053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:00:11.067489 14053 catalog_manager.cc:1347] Generated new cluster ID: f4dd71da544742f093963bbbfe2c6e6b
I20260808 13:00:11.067667 14053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:00:11.086148 14053 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:00:11.087579 14053 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:00:11.096570 14053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: Generated new TSK 0
I20260808 13:00:11.097467 14053 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 13:00:11.180385 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:41391
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:00:11.535181 14073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:11.535533 14073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:11.535712 14073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:11.543519 14073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:11.543700 14073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:11.543876 14073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 13:00:11.554741 14073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:11.556452 14073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:11.557932 14073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:11.572387 14082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:11.572362 14088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:11.572455 14084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:11.573421 14073 server_base.cc:1034] running on GCE node
I20260808 13:00:11.574151 14073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:11.575239 14073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:11.576501 14073 hybrid_clock.cc:648] HybridClock initialized: now 1786194011576448 us; error 47 us; skew 500 ppm
I20260808 13:00:11.576853 14073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:11 dist-test-slave-tk9z krb5kdc[13898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194011, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:11.580349 14073 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 13:00:11.581878 14073 webserver.cc:460] Webserver started at http://127.24.167.1:43833/ using document root <none> and password file <none>
I20260808 13:00:11.582669 14073 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:11.582823 14073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:11.583148 14073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:00:11.586169 14073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "8d651e120b9844df9f45c3fadd207830"
format_stamp: "Formatted at 2026-08-08 13:00:11 on dist-test-slave-tk9z"
server_key: "d72ab587a8e5685b8eb04fffb1eefce7"
server_key_iv: "ff9c738dc4b01fbb84c9f6e1f250537a"
server_key_version: "encryptionkey@0"
I20260808 13:00:11.587088 14073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "8d651e120b9844df9f45c3fadd207830"
format_stamp: "Formatted at 2026-08-08 13:00:11 on dist-test-slave-tk9z"
server_key: "d72ab587a8e5685b8eb04fffb1eefce7"
server_key_iv: "ff9c738dc4b01fbb84c9f6e1f250537a"
server_key_version: "encryptionkey@0"
I20260808 13:00:11.592584 14073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 13:00:11.596212 14098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:11.597766 14073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 13:00:11.597965 14073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "8d651e120b9844df9f45c3fadd207830"
format_stamp: "Formatted at 2026-08-08 13:00:11 on dist-test-slave-tk9z"
server_key: "d72ab587a8e5685b8eb04fffb1eefce7"
server_key_iv: "ff9c738dc4b01fbb84c9f6e1f250537a"
server_key_version: "encryptionkey@0"
I20260808 13:00:11.598174 14073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:11.623808 14073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:11.628360 14073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:11.628679 14073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:11.629674 14073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:00:11.631103 14073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:00:11.631215 14073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:11.631367 14073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:00:11.631443 14073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:11.665809 14073 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:46425
I20260808 13:00:11.665855 14286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:46425 every 8 connection(s)
I20260808 13:00:11.667207 14073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 13:00:11.669971 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 14073
I20260808 13:00:11.670198 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 13:00:11.670966 25244 external_mini_cluster.cc:1428] Setting key fd009fad82cf4271a49a65d59bc4d6cd
Aug 08 13:00:11 dist-test-slave-tk9z krb5kdc[13898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194011, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:00:11.690055 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:11.669903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:56833 (local address 127.24.167.62:41391)
0808 13:00:11.672480 (+ 2577us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:11.672484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:11.672504 (+ 20us) server_negotiation.cc:407] Connection header received
0808 13:00:11.674135 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:11.674144 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:11.674249 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:11.674390 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:11.675938 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:11.677113 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:11.678855 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:11.679610 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:11.679765 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:11.683596 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:11.683604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:11.683612 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:11.683660 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:11.686442 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:11.687422 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:11.687427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:11.687429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:11.687588 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:11.688154 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:11.688158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:11.688161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:11.688477 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:11.688605 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:11.689614 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0808 13:00:11.689804 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2453,"thread_start_us":2338,"threads_started":1}
I20260808 13:00:11.691802 14291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:11.670582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:56833)
0808 13:00:11.671604 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:11.671665 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:11.673838 (+ 2173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:11.674630 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:11.674639 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:11.675167 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:11.675750 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:11.675768 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:11.677298 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:11.677302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:11.678683 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:11.678697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:11.679845 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:11.679850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:11.680077 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:11.680809 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:11.680833 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:11.683325 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:11.686588 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:11.686600 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:11.686622 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:11.687295 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:11.687706 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:11.687714 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:11.687720 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:11.687995 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:11.688637 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:11.688649 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:11.689148 (+ 499us) client_negotiation.cc:769] Sending connection context
0808 13:00:11.689465 (+ 317us) client_negotiation.cc:240] Negotiation successful
0808 13:00:11.690112 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":395,"threads_started":1}
I20260808 13:00:11.693552 14287 heartbeater.cc:347] Connected to a master server at 127.24.167.62:41391
I20260808 13:00:11.693892 14287 heartbeater.cc:464] Registering TS with master...
I20260808 13:00:11.694748 14287 heartbeater.cc:511] Master 127.24.167.62:41391 requested a full tablet report, sending...
I20260808 13:00:11.696940 13971 ts_manager.cc:194] Registered new tserver with Master: 8d651e120b9844df9f45c3fadd207830 (127.24.167.1:46425)
I20260808 13:00:11.698581 13971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:56833
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 13:00:11.735996 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:41391
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:00:12.088423 14297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:12.088755 14297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:12.088929 14297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:12.096560 14297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:12.096735 14297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:12.096931 14297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 13:00:12.107755 14297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:12.109238 14297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:12.110886 14297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:12.125741 14307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:12.125831 14305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:12.125855 14304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:12.126461 14297 server_base.cc:1034] running on GCE node
I20260808 13:00:12.127290 14297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:12.128381 14297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:12.129642 14297 hybrid_clock.cc:648] HybridClock initialized: now 1786194012129570 us; error 67 us; skew 500 ppm
I20260808 13:00:12.130029 14297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:12 dist-test-slave-tk9z krb5kdc[13898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194012, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:12.133607 14297 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 13:00:12.135315 14297 webserver.cc:460] Webserver started at http://127.24.167.2:40007/ using document root <none> and password file <none>
I20260808 13:00:12.136040 14297 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:12.136152 14297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:12.136422 14297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:00:12.139647 14297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "a86eca8fd499441d8228eff286fbe56d"
format_stamp: "Formatted at 2026-08-08 13:00:12 on dist-test-slave-tk9z"
server_key: "1436c608dc9950f505ecba7869b18d95"
server_key_iv: "bd4831dd40d11c60180b79ae552ac3a5"
server_key_version: "encryptionkey@0"
I20260808 13:00:12.140636 14297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "a86eca8fd499441d8228eff286fbe56d"
format_stamp: "Formatted at 2026-08-08 13:00:12 on dist-test-slave-tk9z"
server_key: "1436c608dc9950f505ecba7869b18d95"
server_key_iv: "bd4831dd40d11c60180b79ae552ac3a5"
server_key_version: "encryptionkey@0"
I20260808 13:00:12.146850 14297 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 13:00:12.150693 14317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:12.152305 14297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 13:00:12.152513 14297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "a86eca8fd499441d8228eff286fbe56d"
format_stamp: "Formatted at 2026-08-08 13:00:12 on dist-test-slave-tk9z"
server_key: "1436c608dc9950f505ecba7869b18d95"
server_key_iv: "bd4831dd40d11c60180b79ae552ac3a5"
server_key_version: "encryptionkey@0"
I20260808 13:00:12.152721 14297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:12.169453 14297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:12.173748 14297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:12.174062 14297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:12.175171 14297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:00:12.176582 14297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:00:12.176659 14297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:12.176843 14297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:00:12.176901 14297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:12.213689 14297 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:43141
I20260808 13:00:12.213734 14505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:43141 every 8 connection(s)
I20260808 13:00:12.215216 14297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 13:00:12.215575 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 14297
I20260808 13:00:12.215783 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 13:00:12.216437 25244 external_mini_cluster.cc:1428] Setting key 3e1cec22f6b37adf2fc69052439ba7bf
Aug 08 13:00:12 dist-test-slave-tk9z krb5kdc[13898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194012, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:00:12.241677 14509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:12.218321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:39709 (local address 127.24.167.62:41391)
0808 13:00:12.218891 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:12.218899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:12.220171 (+ 1272us) server_negotiation.cc:407] Connection header received
0808 13:00:12.223401 (+ 3230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:12.223406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:12.223494 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:12.223614 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:12.226045 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.227070 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.229689 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.230181 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.230318 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:12.235568 (+ 5250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:12.235574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:12.235577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:12.235612 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:12.238472 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:12.239227 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:12.239233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:12.239236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:12.239385 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:12.239849 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:12.239857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:12.239861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:12.240147 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:12.240312 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:12.241054 (+ 742us) server_negotiation.cc:299] Negotiation successful
0808 13:00:12.241291 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":339,"threads_started":1}
I20260808 13:00:12.242777 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:12.219057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:39709)
0808 13:00:12.219935 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:12.220030 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:12.222905 (+ 2875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:12.223955 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:12.223970 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:12.224862 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:12.225797 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:12.225828 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.227266 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.227273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:12.229501 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:12.229522 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.230467 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.230477 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:12.230845 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:12.232755 (+ 1910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:12.232790 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:12.235288 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:12.238619 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:12.238629 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:12.238648 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:12.239090 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:12.239513 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:12.239518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:12.239522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:12.239725 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:12.240306 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:12.240323 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:12.240678 (+ 355us) client_negotiation.cc:769] Sending connection context
0808 13:00:12.240977 (+ 299us) client_negotiation.cc:240] Negotiation successful
0808 13:00:12.241561 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":334,"threads_started":1}
I20260808 13:00:12.244446 14506 heartbeater.cc:347] Connected to a master server at 127.24.167.62:41391
I20260808 13:00:12.244781 14506 heartbeater.cc:464] Registering TS with master...
I20260808 13:00:12.245631 14506 heartbeater.cc:511] Master 127.24.167.62:41391 requested a full tablet report, sending...
I20260808 13:00:12.247400 13971 ts_manager.cc:194] Registered new tserver with Master: a86eca8fd499441d8228eff286fbe56d (127.24.167.2:43141)
I20260808 13:00:12.248278 13971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:39709
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 13:00:12.283138 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:41391
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:00:12.636667 14513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:12.637025 14513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:12.637197 14513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:12.644620 14513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:12.644797 14513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:12.644970 14513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 13:00:12.655320 14513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:12.656877 14513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:12.658341 14513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:12.672595 14522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:12.672629 14520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:12.672856 14519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:12.673425 14513 server_base.cc:1034] running on GCE node
I20260808 13:00:12.674026 14513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:12.675074 14513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:12.676290 14513 hybrid_clock.cc:648] HybridClock initialized: now 1786194012676218 us; error 56 us; skew 500 ppm
I20260808 13:00:12.676628 14513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:12 dist-test-slave-tk9z krb5kdc[13898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194012, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:12.680155 14513 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 13:00:12.681684 14513 webserver.cc:460] Webserver started at http://127.24.167.3:37705/ using document root <none> and password file <none>
I20260808 13:00:12.682498 14513 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:12.682642 14513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:12.682986 14513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:00:12.686172 14513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "a727ae0e680643f69a58203d7fd4a06c"
format_stamp: "Formatted at 2026-08-08 13:00:12 on dist-test-slave-tk9z"
server_key: "8a98fc85d8748f1019f7d19274b0d278"
server_key_iv: "ca839cbbd1c7506dce54f48fabedbba7"
server_key_version: "encryptionkey@0"
I20260808 13:00:12.687116 14513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "a727ae0e680643f69a58203d7fd4a06c"
format_stamp: "Formatted at 2026-08-08 13:00:12 on dist-test-slave-tk9z"
server_key: "8a98fc85d8748f1019f7d19274b0d278"
server_key_iv: "ca839cbbd1c7506dce54f48fabedbba7"
server_key_version: "encryptionkey@0"
I20260808 13:00:12.692715 14513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 13:00:12.696739 14539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:12.698334 14513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260808 13:00:12.698585 14513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "a727ae0e680643f69a58203d7fd4a06c"
format_stamp: "Formatted at 2026-08-08 13:00:12 on dist-test-slave-tk9z"
server_key: "8a98fc85d8748f1019f7d19274b0d278"
server_key_iv: "ca839cbbd1c7506dce54f48fabedbba7"
server_key_version: "encryptionkey@0"
I20260808 13:00:12.698828 14513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:12.702240 14287 heartbeater.cc:503] Master 127.24.167.62:41391 was elected leader, sending a full tablet report...
I20260808 13:00:12.735167 14513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:12.739964 14513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:12.740285 14513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:12.741541 14513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:00:12.743335 14513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:00:12.743458 14513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:12.743641 14513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:00:12.743722 14513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:12.780592 14513 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:45997
I20260808 13:00:12.780696 14739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:45997 every 8 connection(s)
I20260808 13:00:12.782048 14513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 13:00:12.782709 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 14513
I20260808 13:00:12.782925 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 13:00:12.783542 25244 external_mini_cluster.cc:1428] Setting key a0b2d6aff25ea53a33ddfbb85e9af852
Aug 08 13:00:12 dist-test-slave-tk9z krb5kdc[13898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194012, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:00:12.804703 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:12.785285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:47383 (local address 127.24.167.62:41391)
0808 13:00:12.785856 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:12.785861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:12.787084 (+ 1223us) server_negotiation.cc:407] Connection header received
0808 13:00:12.789010 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:12.789014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:12.789099 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:12.789199 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:12.790919 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.791731 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.793443 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.793866 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.793986 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:12.797908 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:12.797915 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:12.797920 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:12.797966 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:12.801244 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:12.802162 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:12.802166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:12.802168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:12.802294 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:12.802800 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:12.802803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:12.802805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:12.803108 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:12.803230 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:12.804036 (+ 806us) server_negotiation.cc:299] Negotiation successful
0808 13:00:12.804231 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":274,"threads_started":1}
I20260808 13:00:12.805630 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:12.786085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:47383)
0808 13:00:12.786929 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:12.786974 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:12.788743 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:12.789475 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:12.789483 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:12.790036 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:12.790742 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:12.790762 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.791909 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.791913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:12.793239 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:12.793253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.794112 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.794117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:12.794336 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:12.795186 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:12.795215 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:12.797605 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:12.801411 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:12.801419 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:12.801443 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:12.802035 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:12.802434 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:12.802439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:12.802443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:12.802675 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:12.803213 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:12.803219 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:12.803665 (+ 446us) client_negotiation.cc:769] Sending connection context
0808 13:00:12.803906 (+ 241us) client_negotiation.cc:240] Negotiation successful
0808 13:00:12.804338 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":406,"threads_started":1}
I20260808 13:00:12.807468 14740 heartbeater.cc:347] Connected to a master server at 127.24.167.62:41391
I20260808 13:00:12.807832 14740 heartbeater.cc:464] Registering TS with master...
I20260808 13:00:12.808697 14740 heartbeater.cc:511] Master 127.24.167.62:41391 requested a full tablet report, sending...
I20260808 13:00:12.810587 13971 ts_manager.cc:194] Registered new tserver with Master: a727ae0e680643f69a58203d7fd4a06c (127.24.167.3:45997)
I20260808 13:00:12.811902 13971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:47383
I20260808 13:00:12.813819 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 13:00:12.832679 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:12.818919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.24.167.62:41391)
0808 13:00:12.819240 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:12.819249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:12.819573 (+ 324us) server_negotiation.cc:407] Connection header received
0808 13:00:12.819810 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:12.819814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:12.819883 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:12.820003 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:12.821548 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.822290 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.823791 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.824231 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.824382 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:12.826991 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:12.826997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:12.827000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:12.827036 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:12.829685 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:12.830513 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:12.830519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:12.830523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:12.830669 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:12.831088 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:12.831093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:12.831103 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:12.831454 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:12.831728 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:12.832107 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 13:00:12.832353 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260808 13:00:12.842887 13971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 13:00:12.850230 13971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 13:00:12.882431 14772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:12.864492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:46425 (local address 127.0.0.1:51686)
0808 13:00:12.867248 (+ 2756us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:12.867314 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:12.868669 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:12.870771 (+ 2102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:12.870796 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:12.870843 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:00:12.871389 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:12.871408 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.874171 (+ 2763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.874181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:12.877716 (+ 3535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:12.877752 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.881242 (+ 3490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.881252 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:12.881652 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:12.881709 (+ 57us) client_negotiation.cc:769] Sending connection context
0808 13:00:12.881847 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 13:00:12.882055 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2409,"thread_start_us":15851,"threads_started":1}
I20260808 13:00:12.883101 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:12.864726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51686 (local address 127.24.167.1:46425)
0808 13:00:12.869520 (+ 4794us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:12.869533 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:12.869572 (+ 39us) server_negotiation.cc:407] Connection header received
0808 13:00:12.869704 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:12.869716 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:12.870092 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:12.870364 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:00:12.871715 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.873938 (+ 2223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.878247 (+ 4309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.880890 (+ 2643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.881132 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:12.881424 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:12.882553 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0808 13:00:12.882750 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4532,"thread_start_us":4341,"threads_started":1}
I20260808 13:00:12.888211 14183 tablet_service.cc:1467] Processing CreateTablet for tablet e8667368006949a88b24e11cb63596d6 (DEFAULT_TABLE table=test-table [id=9102f8cced6c454d8b93871efe3a437d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:00:12.889972 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:12.864593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:45997 (local address 127.0.0.1:32962)
0808 13:00:12.879004 (+ 14411us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:12.879032 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:12.879209 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:12.882684 (+ 3475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:12.882689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:12.882720 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:00:12.883112 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:12.883125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.885306 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.885312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:12.887027 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:12.887046 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.888996 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.889003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:12.889176 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:12.889202 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 13:00:12.889330 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 13:00:12.889491 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14210,"mutex_wait_us":27,"thread_start_us":2097,"threads_started":1}
I20260808 13:00:12.890236 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:12.868550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.24.167.2:43141)
0808 13:00:12.881133 (+ 12583us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:12.881143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:12.881175 (+ 32us) server_negotiation.cc:407] Connection header received
0808 13:00:12.881287 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:12.881295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:12.881649 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:12.881914 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:00:12.882829 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.884803 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.887635 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.889126 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.889301 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:12.889485 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:12.889767 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 13:00:12.889958 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12385,"thread_start_us":12264,"threads_started":1}
I20260808 13:00:12.890614 14183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8667368006949a88b24e11cb63596d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:00:12.890707 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:12.865178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.24.167.3:45997)
0808 13:00:12.882012 (+ 16834us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:12.882020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:12.882053 (+ 33us) server_negotiation.cc:407] Connection header received
0808 13:00:12.882155 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:12.882162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:12.882438 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:12.883321 (+ 883us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:00:12.883623 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.885158 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.887207 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.888665 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.888806 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:12.889057 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:12.890102 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0808 13:00:12.890307 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16641,"thread_start_us":16447,"threads_started":1}
I20260808 13:00:12.891884 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:12.868357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:43141 (local address 127.0.0.1:43174)
0808 13:00:12.880771 (+ 12414us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:12.880797 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:12.880936 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:12.882177 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:12.882182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:12.882208 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:00:12.882631 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:12.882648 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.884970 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.884975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:12.887419 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:12.887435 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:12.889368 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:12.889373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:12.889542 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:12.889572 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 13:00:12.889665 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 13:00:12.889797 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12253,"mutex_wait_us":143,"thread_start_us":10960,"threads_started":1}
I20260808 13:00:12.894598 14625 tablet_service.cc:1467] Processing CreateTablet for tablet e8667368006949a88b24e11cb63596d6 (DEFAULT_TABLE table=test-table [id=9102f8cced6c454d8b93871efe3a437d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:00:12.895902 14625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8667368006949a88b24e11cb63596d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:00:12.895599 14396 tablet_service.cc:1467] Processing CreateTablet for tablet e8667368006949a88b24e11cb63596d6 (DEFAULT_TABLE table=test-table [id=9102f8cced6c454d8b93871efe3a437d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:00:12.897040 14396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8667368006949a88b24e11cb63596d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:00:12.909416 14786 tablet_bootstrap.cc:492] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c: Bootstrap starting.
I20260808 13:00:12.911018 14782 tablet_bootstrap.cc:492] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830: Bootstrap starting.
I20260808 13:00:12.914148 14786 tablet_bootstrap.cc:654] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c: Neither blocks nor log segments found. Creating new log.
I20260808 13:00:12.916134 14786 log.cc:824] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:12.916394 14782 tablet_bootstrap.cc:654] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830: Neither blocks nor log segments found. Creating new log.
I20260808 13:00:12.916767 14788 tablet_bootstrap.cc:492] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d: Bootstrap starting.
I20260808 13:00:12.917872 14782 log.cc:824] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:12.920516 14788 tablet_bootstrap.cc:654] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d: Neither blocks nor log segments found. Creating new log.
I20260808 13:00:12.921211 14786 tablet_bootstrap.cc:492] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c: No bootstrap required, opened a new log
I20260808 13:00:12.921537 14786 ts_tablet_manager.cc:1397] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260808 13:00:12.921576 14788 log.cc:824] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:12.924578 14782 tablet_bootstrap.cc:492] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830: No bootstrap required, opened a new log
I20260808 13:00:12.924679 14788 tablet_bootstrap.cc:492] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d: No bootstrap required, opened a new log
I20260808 13:00:12.924899 14782 ts_tablet_manager.cc:1397] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.007s
I20260808 13:00:12.924939 14788 ts_tablet_manager.cc:1397] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 13:00:12.932391 14782 raft_consensus.cc:348] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:12.932847 14782 raft_consensus.cc:374] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:00:12.932602 14786 raft_consensus.cc:348] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:12.932999 14782 raft_consensus.cc:729] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d651e120b9844df9f45c3fadd207830, State: Initialized, Role: FOLLOWER
I20260808 13:00:12.933112 14786 raft_consensus.cc:374] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:00:12.933216 14786 raft_consensus.cc:729] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a727ae0e680643f69a58203d7fd4a06c, State: Initialized, Role: FOLLOWER
I20260808 13:00:12.933748 14782 consensus_queue.cc:260] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:12.933939 14786 consensus_queue.cc:260] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:12.936414 14788 raft_consensus.cc:348] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:12.937101 14788 raft_consensus.cc:374] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:00:12.937242 14788 raft_consensus.cc:729] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a86eca8fd499441d8228eff286fbe56d, State: Initialized, Role: FOLLOWER
I20260808 13:00:12.937850 14782 ts_tablet_manager.cc:1428] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830: Time spent starting tablet: real 0.013s user 0.000s sys 0.011s
I20260808 13:00:12.938207 14788 consensus_queue.cc:260] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:12.938957 14740 heartbeater.cc:503] Master 127.24.167.62:41391 was elected leader, sending a full tablet report...
I20260808 13:00:12.941545 14786 ts_tablet_manager.cc:1428] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c: Time spent starting tablet: real 0.020s user 0.014s sys 0.005s
I20260808 13:00:12.941898 14506 heartbeater.cc:503] Master 127.24.167.62:41391 was elected leader, sending a full tablet report...
I20260808 13:00:12.942564 14788 ts_tablet_manager.cc:1428] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d: Time spent starting tablet: real 0.018s user 0.014s sys 0.005s
W20260808 13:00:12.967588 14507 tablet.cc:2369] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 13:00:13.034255 14747 tablet.cc:2369] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:00:13.069876 14797 raft_consensus.cc:482] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 13:00:13.070284 14797 raft_consensus.cc:504] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:13.072263 14797 leader_election.cc:288] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d651e120b9844df9f45c3fadd207830 (127.24.167.1:46425), a727ae0e680643f69a58203d7fd4a06c (127.24.167.3:45997)
I20260808 13:00:13.080286 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:13.072973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:40695 (local address 127.24.167.1:46425)
0808 13:00:13.073247 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:13.073256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:13.073846 (+ 590us) server_negotiation.cc:407] Connection header received
0808 13:00:13.074034 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:13.074041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:13.074117 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:13.074215 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:00:13.075006 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.075817 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.078364 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.079238 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.079444 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:13.079533 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:13.079864 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 13:00:13.080002 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 13:00:13.080287 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:13.072765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:57567 (local address 127.24.167.3:45997)
0808 13:00:13.073000 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:13.073008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:13.074038 (+ 1030us) server_negotiation.cc:407] Connection header received
0808 13:00:13.074275 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:13.074280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:13.074364 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:13.074487 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:00:13.075176 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.075910 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.078438 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.079226 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.079413 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:13.079518 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:13.079862 (+ 344us) server_negotiation.cc:299] Negotiation successful
0808 13:00:13.080010 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 13:00:13.080508 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:13.072793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:46425 (local address 127.24.167.2:40695)
0808 13:00:13.073718 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:13.073735 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:13.073879 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:13.074267 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:13.074272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:13.074300 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:00:13.074836 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:13.074848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.075962 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.075971 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:13.078191 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:13.078204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.079541 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.079549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:13.079696 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:13.079720 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 13:00:13.079833 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 13:00:13.079988 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":162,"spinlock_wait_cycles":13568,"thread_start_us":542,"threads_started":1}
I20260808 13:00:13.080530 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:13.072832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:45997 (local address 127.24.167.2:57567)
0808 13:00:13.073901 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:13.073923 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:13.074094 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:13.074516 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:13.074519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:13.074570 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:00:13.075011 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:13.075022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.076058 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.076062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:13.078196 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:13.078209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.079510 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.079520 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:13.079685 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:13.079717 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 13:00:13.079833 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 13:00:13.079985 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":46,"spinlock_wait_cycles":3328,"thread_start_us":679,"threads_started":1}
I20260808 13:00:13.081723 14661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8667368006949a88b24e11cb63596d6" candidate_uuid: "a86eca8fd499441d8228eff286fbe56d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a727ae0e680643f69a58203d7fd4a06c" is_pre_election: true
I20260808 13:00:13.081825 14210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8667368006949a88b24e11cb63596d6" candidate_uuid: "a86eca8fd499441d8228eff286fbe56d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d651e120b9844df9f45c3fadd207830" is_pre_election: true
I20260808 13:00:13.082346 14661 raft_consensus.cc:2454] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a86eca8fd499441d8228eff286fbe56d in term 0.
I20260808 13:00:13.082484 14210 raft_consensus.cc:2454] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a86eca8fd499441d8228eff286fbe56d in term 0.
I20260808 13:00:13.083153 14322 leader_election.cc:302] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8d651e120b9844df9f45c3fadd207830, a727ae0e680643f69a58203d7fd4a06c, a86eca8fd499441d8228eff286fbe56d; no voters:
I20260808 13:00:13.083621 14797 raft_consensus.cc:2781] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 13:00:13.083777 14797 raft_consensus.cc:482] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 13:00:13.083916 14797 raft_consensus.cc:3037] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:00:13.086198 14797 raft_consensus.cc:504] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:13.086959 14797 leader_election.cc:288] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [CANDIDATE]: Term 1 election: Requested vote from peers 8d651e120b9844df9f45c3fadd207830 (127.24.167.1:46425), a727ae0e680643f69a58203d7fd4a06c (127.24.167.3:45997)
I20260808 13:00:13.087528 14210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8667368006949a88b24e11cb63596d6" candidate_uuid: "a86eca8fd499441d8228eff286fbe56d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d651e120b9844df9f45c3fadd207830"
I20260808 13:00:13.087527 14661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8667368006949a88b24e11cb63596d6" candidate_uuid: "a86eca8fd499441d8228eff286fbe56d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a727ae0e680643f69a58203d7fd4a06c"
I20260808 13:00:13.087776 14661 raft_consensus.cc:3037] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:00:13.087779 14210 raft_consensus.cc:3037] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:00:13.091293 14210 raft_consensus.cc:2454] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a86eca8fd499441d8228eff286fbe56d in term 1.
I20260808 13:00:13.091293 14661 raft_consensus.cc:2454] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a86eca8fd499441d8228eff286fbe56d in term 1.
I20260808 13:00:13.091848 14321 leader_election.cc:302] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d651e120b9844df9f45c3fadd207830, a86eca8fd499441d8228eff286fbe56d; no voters:
I20260808 13:00:13.092223 14797 raft_consensus.cc:2781] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:00:13.092721 14797 raft_consensus.cc:686] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [term 1 LEADER]: Becoming Leader. State: Replica: a86eca8fd499441d8228eff286fbe56d, State: Running, Role: LEADER
I20260808 13:00:13.093225 14797 consensus_queue.cc:237] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:13.098134 13969 catalog_manager.cc:5495] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d reported cstate change: term changed from 0 to 1, leader changed from <none> to a86eca8fd499441d8228eff286fbe56d (127.24.167.2). New cstate: current_term: 1 leader_uuid: "a86eca8fd499441d8228eff286fbe56d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } health_report { overall_health: HEALTHY } } }
I20260808 13:00:13.136360 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:13.130927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.24.167.2:43141)
0808 13:00:13.131115 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:13.131119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:13.131137 (+ 18us) server_negotiation.cc:407] Connection header received
0808 13:00:13.131229 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:13.131233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:13.131319 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:13.131447 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:00:13.131823 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.132554 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.134182 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.134679 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.134775 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:13.135272 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:00:13.135792 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:00:13.135902 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:13.136002 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 13:00:13.136073 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 13:00:13.145934 14661 raft_consensus.cc:1261] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c [term 1 FOLLOWER]: Refusing update from remote peer a86eca8fd499441d8228eff286fbe56d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:00:13.145999 14210 raft_consensus.cc:1261] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830 [term 1 FOLLOWER]: Refusing update from remote peer a86eca8fd499441d8228eff286fbe56d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:00:13.147095 14801 consensus_queue.cc:1035] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:00:13.147492 14797 consensus_queue.cc:1035] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:00:13.178159 13966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:00:13.178568 13966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:00:13.182719 13966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: Sending DeleteTablet for 3 replicas of tablet e8667368006949a88b24e11cb63596d6
I20260808 13:00:13.184257 14183 tablet_service.cc:1514] Processing DeleteTablet for tablet e8667368006949a88b24e11cb63596d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:00:13 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:51686
I20260808 13:00:13.184258 14396 tablet_service.cc:1514] Processing DeleteTablet for tablet e8667368006949a88b24e11cb63596d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:00:13 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:43174
I20260808 13:00:13.185626 14625 tablet_service.cc:1514] Processing DeleteTablet for tablet e8667368006949a88b24e11cb63596d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:00:13 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:32962
I20260808 13:00:13.186017 14817 tablet_replica.cc:330] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830: stopping tablet replica
I20260808 13:00:13.186436 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 14073
I20260808 13:00:13.186503 14817 raft_consensus.cc:2229] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:00:13.187049 14817 raft_consensus.cc:2258] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:00:13.187779 14818 tablet_replica.cc:330] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d: stopping tablet replica
I20260808 13:00:13.188169 14818 raft_consensus.cc:2229] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [term 1 LEADER]: Raft consensus shutting down.
I20260808 13:00:13.188982 14818 raft_consensus.cc:2258] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:00:13.190753 14817 ts_tablet_manager.cc:1905] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:00:13.194188 14818 ts_tablet_manager.cc:1905] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:00:13.198448 14817 ts_tablet_manager.cc:1918] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:00:13.198695 14817 log.cc:1196] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/e8667368006949a88b24e11cb63596d6
I20260808 13:00:13.199316 14817 ts_tablet_manager.cc:1939] T e8667368006949a88b24e11cb63596d6 P 8d651e120b9844df9f45c3fadd207830: Deleting consensus metadata
I20260808 13:00:13.201503 13944 catalog_manager.cc:4841] TS 8d651e120b9844df9f45c3fadd207830 (127.24.167.1:46425): tablet e8667368006949a88b24e11cb63596d6 (table test-table [id=9102f8cced6c454d8b93871efe3a437d]) successfully deleted
I20260808 13:00:13.201607 14819 tablet_replica.cc:330] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c: stopping tablet replica
I20260808 13:00:13.202176 14819 raft_consensus.cc:2229] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:00:13.202932 14819 raft_consensus.cc:2258] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:00:13.205753 14819 ts_tablet_manager.cc:1905] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:00:13.207607 14818 ts_tablet_manager.cc:1918] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:00:13.207863 14818 log.cc:1196] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/e8667368006949a88b24e11cb63596d6
I20260808 13:00:13.208537 14818 ts_tablet_manager.cc:1939] T e8667368006949a88b24e11cb63596d6 P a86eca8fd499441d8228eff286fbe56d: Deleting consensus metadata
I20260808 13:00:13.210610 13942 catalog_manager.cc:4841] TS a86eca8fd499441d8228eff286fbe56d (127.24.167.2:43141): tablet e8667368006949a88b24e11cb63596d6 (table test-table [id=9102f8cced6c454d8b93871efe3a437d]) successfully deleted
Aug 08 13:00:13 dist-test-slave-tk9z krb5kdc[13898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 13:00:13.214615 14819 ts_tablet_manager.cc:1918] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:00:13.214932 14819 log.cc:1196] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/e8667368006949a88b24e11cb63596d6
I20260808 13:00:13.215688 14819 ts_tablet_manager.cc:1939] T e8667368006949a88b24e11cb63596d6 P a727ae0e680643f69a58203d7fd4a06c: Deleting consensus metadata
I20260808 13:00:13.218201 13945 catalog_manager.cc:4841] TS a727ae0e680643f69a58203d7fd4a06c (127.24.167.3:45997): tablet e8667368006949a88b24e11cb63596d6 (table test-table [id=9102f8cced6c454d8b93871efe3a437d]) successfully deleted
I20260808 13:00:13.222939 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:13.187915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51694 (local address 127.24.167.1:46425)
0808 13:00:13.188136 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:13.188147 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:13.197924 (+ 9777us) server_negotiation.cc:407] Connection header received
0808 13:00:13.198548 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:13.198555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:13.198650 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:13.198773 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:13.203252 (+ 4479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.204549 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.206871 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.207620 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.207740 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:13.214634 (+ 6894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:13.214659 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:13.214681 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:13.214771 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:13.219397 (+ 4626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:13.220303 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:13.220311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:13.220314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:13.220477 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:13.220958 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:13.220963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:13.220965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:13.221458 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:13.221842 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:13.222191 (+ 349us) server_negotiation.cc:299] Negotiation successful
0808 13:00:13.222498 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 13:00:13.223793 14281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:13.319799 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 14073
I20260808 13:00:13.353036 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 14297
Aug 08 13:00:13 dist-test-slave-tk9z krb5kdc[13898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 13:00:13.372282 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:13.353920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.24.167.2:43141)
0808 13:00:13.354124 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:13.354128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:13.354147 (+ 19us) server_negotiation.cc:407] Connection header received
0808 13:00:13.354347 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:13.354351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:13.354485 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:13.354612 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:13.356329 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.357374 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.359371 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.359809 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.359910 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:13.365632 (+ 5722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:13.365651 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:13.365669 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:13.365721 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:13.369116 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:13.370062 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:13.370070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:13.370073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:13.370232 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:13.370712 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:13.370716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:13.370718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:13.371138 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:13.371395 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:13.371757 (+ 362us) server_negotiation.cc:299] Negotiation successful
0808 13:00:13.371960 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 13:00:13.373095 14499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:13.489605 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 14297
I20260808 13:00:13.516239 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 14513
Aug 08 13:00:13 dist-test-slave-tk9z krb5kdc[13898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 13:00:13.532269 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:13.517056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.24.167.3:45997)
0808 13:00:13.517288 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:13.517293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:13.517312 (+ 19us) server_negotiation.cc:407] Connection header received
0808 13:00:13.517464 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:13.517468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:13.517531 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:13.517616 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:13.519062 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.519774 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.521054 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:13.521489 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:13.521585 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:13.526240 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:13.526254 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:13.526272 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:13.526314 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:13.529282 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:13.530161 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:13.530169 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:13.530171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:13.530341 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:13.530832 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:13.530836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:13.530838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:13.531227 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:13.531457 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:13.531686 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 13:00:13.531854 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 13:00:13.532938 14734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:13.624198 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 14513
I20260808 13:00:13.648947 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 13920
I20260808 13:00:13.650188 14026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:13.767251 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 13920
I20260808 13:00:13.799023 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:41391
--webserver_interface=127.24.167.62
--webserver_port=36235
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:41391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:00:14.154035 14828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:14.154345 14828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:14.154588 14828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:14.161845 14828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:00:14.161958 14828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:14.162060 14828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:14.162163 14828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:00:14.162246 14828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:00:14.172482 14828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:41391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=36235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:14.173866 14828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:14.175263 14828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:14.203230 14840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:14.203203 14841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:14.203591 14849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:14.205150 14828 server_base.cc:1034] running on GCE node
I20260808 13:00:14.206103 14828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:14.207962 14828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:14.209214 14828 hybrid_clock.cc:648] HybridClock initialized: now 1786194014209151 us; error 55 us; skew 500 ppm
I20260808 13:00:14.209648 14828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:14 dist-test-slave-tk9z krb5kdc[13898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194014, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:14.213403 14828 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:00:14.215369 14828 webserver.cc:460] Webserver started at http://127.24.167.62:36235/ using document root <none> and password file <none>
I20260808 13:00:14.216135 14828 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:14.216264 14828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:14.222827 14828 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 13:00:14.225939 14859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:14.227597 14828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:00:14.227804 14828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "0520edffaf2a4ee1ac7a31a6011424ae"
format_stamp: "Formatted at 2026-08-08 13:00:10 on dist-test-slave-tk9z"
server_key: "c16da6c002060beaa75c4da425a758fc"
server_key_iv: "cd0408edbd7004997948db7f8ee18468"
server_key_version: "encryptionkey@0"
I20260808 13:00:14.228969 14828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:14.247606 14828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:14.251996 14828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:14.252312 14828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:14.272531 14828 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:41391
I20260808 13:00:14.272550 14941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:41391 every 8 connection(s)
I20260808 13:00:14.274154 14828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:00:14.278362 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 14828
I20260808 13:00:14.279292 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:46425
--local_ip_for_outbound_sockets=127.24.167.1
--tserver_master_addrs=127.24.167.62:41391
--webserver_port=43833
--webserver_interface=127.24.167.1
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 13:00:14.281538 14943 sys_catalog.cc:263] Verifying existing consensus state
I20260808 13:00:14.285063 14943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: Bootstrap starting.
I20260808 13:00:14.305316 14943 log.cc:824] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:14.318879 14943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 13:00:14.319474 14943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: Bootstrap complete.
I20260808 13:00:14.329249 14943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } }
I20260808 13:00:14.330175 14943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0520edffaf2a4ee1ac7a31a6011424ae, State: Initialized, Role: FOLLOWER
I20260808 13:00:14.330883 14943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } }
I20260808 13:00:14.331161 14943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:00:14.331305 14943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:00:14.331509 14943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 1 FOLLOWER]: Advancing to term 2
I20260808 13:00:14.335863 14943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } }
I20260808 13:00:14.336462 14943 leader_election.cc:302] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0520edffaf2a4ee1ac7a31a6011424ae; no voters:
I20260808 13:00:14.337196 14943 leader_election.cc:288] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 13:00:14.337337 14947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 2 FOLLOWER]: Leader election won for term 2
I20260808 13:00:14.339911 14943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:00:14.344512 14947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 2 LEADER]: Becoming Leader. State: Replica: 0520edffaf2a4ee1ac7a31a6011424ae, State: Running, Role: LEADER
I20260808 13:00:14.345389 14947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } }
I20260808 13:00:14.348544 14948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } } }
I20260808 13:00:14.348946 14948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [sys.catalog]: This master's current role is: LEADER
I20260808 13:00:14.348757 14951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0520edffaf2a4ee1ac7a31a6011424ae. Latest consensus state: current_term: 2 leader_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } } }
I20260808 13:00:14.349298 14951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [sys.catalog]: This master's current role is: LEADER
I20260808 13:00:14.360106 14975 catalog_manager.cc:1259] Loaded cluster ID: f4dd71da544742f093963bbbfe2c6e6b
I20260808 13:00:14.360225 14975 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: loading cluster ID for follower catalog manager: success
W20260808 13:00:14.362273 14975 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260808 13:00:14.362453 14975 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260808 13:00:14.362687 14970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:00:14.367604 14970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:00:14.368911 14970 catalog_manager.cc:1259] Loaded cluster ID: f4dd71da544742f093963bbbfe2c6e6b
I20260808 13:00:14.369024 14970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:00:14.371901 14970 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 13:00:14.389315 14970 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 13:00:14.399703 14970 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:00:14.401621 14970 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: Loaded TSK: 0
I20260808 13:00:14.403268 14970 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 13:00:14.646205 14945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:14.646684 14945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:14.646876 14945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:14.655272 14945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:14.655449 14945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:14.655627 14945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 13:00:14.665980 14945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:46425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=43833
--tserver_master_addrs=127.24.167.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:14.667466 14945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:14.668898 14945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:14.684898 14986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:14.684870 14987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:14.685475 14991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:14.686524 14945 server_base.cc:1034] running on GCE node
I20260808 13:00:14.687234 14945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:14.688290 14945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:14.689532 14945 hybrid_clock.cc:648] HybridClock initialized: now 1786194014689467 us; error 58 us; skew 500 ppm
I20260808 13:00:14.689877 14945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:14 dist-test-slave-tk9z krb5kdc[13898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194014, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:14.693235 14945 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 13:00:14.694983 14945 webserver.cc:460] Webserver started at http://127.24.167.1:43833/ using document root <none> and password file <none>
I20260808 13:00:14.695703 14945 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:14.695856 14945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:14.701746 14945 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 13:00:14.705003 14999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:14.706717 14945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:00:14.706918 14945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "8d651e120b9844df9f45c3fadd207830"
format_stamp: "Formatted at 2026-08-08 13:00:11 on dist-test-slave-tk9z"
server_key: "d72ab587a8e5685b8eb04fffb1eefce7"
server_key_iv: "ff9c738dc4b01fbb84c9f6e1f250537a"
server_key_version: "encryptionkey@0"
I20260808 13:00:14.708092 14945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:14.734191 14945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:14.738881 14945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:14.739171 14945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:14.740319 14945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:00:14.741915 14945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:00:14.742050 14945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:14.742189 14945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:00:14.742280 14945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:14.778982 14945 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:46425
I20260808 13:00:14.779078 15170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:46425 every 8 connection(s)
I20260808 13:00:14.780548 14945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 13:00:14.788957 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 14945
I20260808 13:00:14.789817 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:43141
--local_ip_for_outbound_sockets=127.24.167.2
--tserver_master_addrs=127.24.167.62:41391
--webserver_port=40007
--webserver_interface=127.24.167.2
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 13:00:14 dist-test-slave-tk9z krb5kdc[13898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194014, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:00:14.807600 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:14.783644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:57749 (local address 127.24.167.62:41391)
0808 13:00:14.784299 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:14.784312 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:14.784802 (+ 490us) server_negotiation.cc:407] Connection header received
0808 13:00:14.787117 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:14.787130 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:14.787494 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:14.787825 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:14.789487 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:14.791735 (+ 2248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:14.793471 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:14.794609 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:14.794835 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:14.798747 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:14.798761 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:14.798786 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:14.798860 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:14.802431 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:14.803494 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:14.803501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:14.803506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:14.803677 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:14.804213 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:14.804216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:14.804218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:14.804847 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:14.805027 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:14.806054 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0808 13:00:14.806466 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":307,"threads_started":1}
I20260808 13:00:14.807885 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:14.783669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:57749)
0808 13:00:14.784593 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:14.784647 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:14.786538 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:14.788007 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:14.788015 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:14.788602 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:14.789224 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:14.789257 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:14.791959 (+ 2702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:14.791963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:14.793264 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:14.793320 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:14.794871 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:14.794877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:14.795086 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:14.795824 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:14.795854 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:14.798433 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:14.802603 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:14.802617 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:14.802636 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:14.803343 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:14.803807 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:14.803815 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:14.803821 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:14.804101 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:14.805013 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:14.805024 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:14.805488 (+ 464us) client_negotiation.cc:769] Sending connection context
0808 13:00:14.805858 (+ 370us) client_negotiation.cc:240] Negotiation successful
0808 13:00:14.806528 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":371,"threads_started":1}
I20260808 13:00:14.811456 15172 heartbeater.cc:347] Connected to a master server at 127.24.167.62:41391
I20260808 13:00:14.811816 15172 heartbeater.cc:464] Registering TS with master...
I20260808 13:00:14.812721 15172 heartbeater.cc:511] Master 127.24.167.62:41391 requested a full tablet report, sending...
I20260808 13:00:14.815663 14887 ts_manager.cc:194] Registered new tserver with Master: 8d651e120b9844df9f45c3fadd207830 (127.24.167.1:46425)
I20260808 13:00:14.818092 14887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:57749
W20260808 13:00:15.151175 15183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:15.151477 15183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:15.151575 15183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:15.158766 15183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:15.158883 15183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:15.158998 15183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 13:00:15.168875 15183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:43141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=40007
--tserver_master_addrs=127.24.167.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:15.170181 15183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:15.171576 15183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:15.185021 15196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:15.184944 15197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:15.184990 15199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:15.186245 15183 server_base.cc:1034] running on GCE node
I20260808 13:00:15.186950 15183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:15.187971 15183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:15.189181 15183 hybrid_clock.cc:648] HybridClock initialized: now 1786194015189131 us; error 47 us; skew 500 ppm
I20260808 13:00:15.189513 15183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:15 dist-test-slave-tk9z krb5kdc[13898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194015, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:15.192911 15183 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 13:00:15.194458 15183 webserver.cc:460] Webserver started at http://127.24.167.2:40007/ using document root <none> and password file <none>
I20260808 13:00:15.195180 15183 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:15.195348 15183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:15.200770 15183 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 13:00:15.203583 15210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:15.205027 15183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:00:15.205211 15183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "a86eca8fd499441d8228eff286fbe56d"
format_stamp: "Formatted at 2026-08-08 13:00:12 on dist-test-slave-tk9z"
server_key: "1436c608dc9950f505ecba7869b18d95"
server_key_iv: "bd4831dd40d11c60180b79ae552ac3a5"
server_key_version: "encryptionkey@0"
I20260808 13:00:15.206300 15183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:15.223893 15183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:15.228328 15183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:15.228631 15183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:15.229745 15183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:00:15.231261 15183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:00:15.231389 15183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:15.231525 15183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:00:15.231618 15183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:15.266144 15183 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:43141
I20260808 13:00:15.266227 15375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:43141 every 8 connection(s)
I20260808 13:00:15.267803 15183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 13:00:15.273032 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 15183
I20260808 13:00:15.273707 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:45997
--local_ip_for_outbound_sockets=127.24.167.3
--tserver_master_addrs=127.24.167.62:41391
--webserver_port=37705
--webserver_interface=127.24.167.3
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 13:00:15 dist-test-slave-tk9z krb5kdc[13898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194015, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:00:15.292891 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:15.270869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:56591 (local address 127.24.167.62:41391)
0808 13:00:15.271060 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:15.271064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:15.272483 (+ 1419us) server_negotiation.cc:407] Connection header received
0808 13:00:15.275205 (+ 2722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:15.275214 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:15.275344 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:15.275472 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:15.277551 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.278769 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.281342 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.282051 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.282177 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:15.286462 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:15.286467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:15.286471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:15.286508 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:15.289391 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:15.290213 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:15.290219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:15.290225 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:15.290455 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:15.290916 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:15.290921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:15.290925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:15.291237 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:15.291397 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:15.292231 (+ 834us) server_negotiation.cc:299] Negotiation successful
0808 13:00:15.292507 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 13:00:15.294061 15381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:15.271290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:56591)
0808 13:00:15.272281 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:15.272335 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:15.274865 (+ 2530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:15.275767 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:15.275783 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:15.276542 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:15.277339 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:15.277363 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.278974 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.278983 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:15.281155 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:15.281173 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.282464 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.282476 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:15.282819 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:15.283685 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:15.283721 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:15.286144 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:15.289582 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:15.289591 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:15.289609 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:15.290073 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:15.290576 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:15.290581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:15.290585 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:15.290797 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:15.291382 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:15.291389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:15.291829 (+ 440us) client_negotiation.cc:769] Sending connection context
0808 13:00:15.292121 (+ 292us) client_negotiation.cc:240] Negotiation successful
0808 13:00:15.292719 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":415,"threads_started":1}
I20260808 13:00:15.295691 15377 heartbeater.cc:347] Connected to a master server at 127.24.167.62:41391
I20260808 13:00:15.296067 15377 heartbeater.cc:464] Registering TS with master...
I20260808 13:00:15.296870 15377 heartbeater.cc:511] Master 127.24.167.62:41391 requested a full tablet report, sending...
I20260808 13:00:15.298516 14887 ts_manager.cc:194] Registered new tserver with Master: a86eca8fd499441d8228eff286fbe56d (127.24.167.2:43141)
I20260808 13:00:15.299504 14887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:56591
W20260808 13:00:15.625564 15382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:15.625865 15382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:15.626046 15382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:15.633461 15382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:15.633592 15382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:15.633777 15382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 13:00:15.643872 15382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:45997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=37705
--tserver_master_addrs=127.24.167.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:15.645349 15382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:15.646844 15382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:15.660362 15389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:15.660386 15394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:15.660310 15390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:15.661468 15382 server_base.cc:1034] running on GCE node
I20260808 13:00:15.662106 15382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:15.663162 15382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:15.664400 15382 hybrid_clock.cc:648] HybridClock initialized: now 1786194015664338 us; error 56 us; skew 500 ppm
I20260808 13:00:15.664731 15382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:15 dist-test-slave-tk9z krb5kdc[13898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194015, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:15.671367 15382 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 13:00:15.673094 15382 webserver.cc:460] Webserver started at http://127.24.167.3:37705/ using document root <none> and password file <none>
I20260808 13:00:15.673879 15382 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:15.674065 15382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:15.679893 15382 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 13:00:15.682915 15406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:15.684324 15382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 13:00:15.684522 15382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "a727ae0e680643f69a58203d7fd4a06c"
format_stamp: "Formatted at 2026-08-08 13:00:12 on dist-test-slave-tk9z"
server_key: "8a98fc85d8748f1019f7d19274b0d278"
server_key_iv: "ca839cbbd1c7506dce54f48fabedbba7"
server_key_version: "encryptionkey@0"
I20260808 13:00:15.685602 15382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:15.714789 15382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:15.719494 15382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:15.719794 15382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:15.720844 15382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:00:15.722262 15382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:00:15.722404 15382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:15.722543 15382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:00:15.722635 15382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:15.758661 15382 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:45997
I20260808 13:00:15.758755 15593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:45997 every 8 connection(s)
I20260808 13:00:15.760211 15382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 13:00:15.768442 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 15382
Aug 08 13:00:15 dist-test-slave-tk9z krb5kdc[13898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194015, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:00:15.783138 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:15.762963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:38619 (local address 127.24.167.62:41391)
0808 13:00:15.763161 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:15.763166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:15.764334 (+ 1168us) server_negotiation.cc:407] Connection header received
0808 13:00:15.766155 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:15.766160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:15.766230 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:15.766322 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:15.768084 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.769191 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.771479 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.771925 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.772068 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:15.775897 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:15.775902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:15.775905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:15.775943 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:15.779078 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:15.780199 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:15.780206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:15.780211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:15.780423 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:15.781034 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:15.781041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:15.781045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:15.781450 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:15.781609 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:15.782561 (+ 952us) server_negotiation.cc:299] Negotiation successful
0808 13:00:15.782824 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 13:00:15.784412 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:15.763470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:38619)
0808 13:00:15.764169 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:15.764214 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:15.765893 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:15.766715 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:15.766725 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:15.767293 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:15.767872 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:15.767890 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.769379 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.769383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:15.771301 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:15.771316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.772179 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.772187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:15.772442 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:15.773197 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:15.773222 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:15.775642 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:15.779257 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:15.779268 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:15.779289 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:15.780018 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:15.780579 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:15.780584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:15.780587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:15.780861 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:15.781619 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:15.781627 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:15.782059 (+ 432us) client_negotiation.cc:769] Sending connection context
0808 13:00:15.782402 (+ 343us) client_negotiation.cc:240] Negotiation successful
0808 13:00:15.782997 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":283,"threads_started":1}
I20260808 13:00:15.786002 15594 heartbeater.cc:347] Connected to a master server at 127.24.167.62:41391
I20260808 13:00:15.786414 15594 heartbeater.cc:464] Registering TS with master...
I20260808 13:00:15.787410 15594 heartbeater.cc:511] Master 127.24.167.62:41391 requested a full tablet report, sending...
I20260808 13:00:15.787870 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:15.769757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.24.167.62:41391)
0808 13:00:15.770751 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:15.770756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:15.770779 (+ 23us) server_negotiation.cc:407] Connection header received
0808 13:00:15.770827 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:15.770830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:15.770898 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:15.771027 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:15.772667 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.773584 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.775143 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.775621 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.775775 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:15.779420 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:15.779458 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:15.779465 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:15.779505 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:15.782824 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:15.785641 (+ 2817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:15.785646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:15.785649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:15.785793 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:15.786245 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:15.786249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:15.786251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:15.786632 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:15.786826 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:15.787264 (+ 438us) server_negotiation.cc:299] Negotiation successful
0808 13:00:15.787475 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":783,"threads_started":1}
I20260808 13:00:15.789172 14887 ts_manager.cc:194] Registered new tserver with Master: a727ae0e680643f69a58203d7fd4a06c (127.24.167.3:45997)
I20260808 13:00:15.790535 14887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:38619
I20260808 13:00:15.790802 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 13:00:15.798434 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:15.791883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.24.167.62:41391)
0808 13:00:15.792126 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:15.792134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:15.792526 (+ 392us) server_negotiation.cc:407] Connection header received
0808 13:00:15.792748 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:15.792752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:15.792876 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:15.792999 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:00:15.793549 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.794221 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.795933 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.796646 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.796763 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:15.797231 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:00:15.797757 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:00:15.797875 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:15.797959 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 13:00:15.798026 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 13:00:15.799480 14887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 13:00:15.807173 14887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 13:00:15.822408 15172 heartbeater.cc:503] Master 127.24.167.62:41391 was elected leader, sending a full tablet report...
I20260808 13:00:15.845114 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:15.821378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.24.167.1:46425)
0808 13:00:15.837135 (+ 15757us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:15.837145 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:15.837176 (+ 31us) server_negotiation.cc:407] Connection header received
0808 13:00:15.837291 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:15.837301 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:15.837538 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:15.837744 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:00:15.838604 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.840207 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.842722 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.843813 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.843982 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:15.844273 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:15.844592 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 13:00:15.844770 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15545,"thread_start_us":15372,"threads_started":1}
I20260808 13:00:15.845228 15611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:15.820011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:46425 (local address 127.0.0.1:51696)
0808 13:00:15.835174 (+ 15163us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:15.835250 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:15.835528 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:15.837761 (+ 2233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:15.837770 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:15.837807 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:00:15.838330 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:15.838343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.840364 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.840371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:15.842518 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:15.842532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.844061 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.844068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:15.844339 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:15.844389 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 13:00:15.844529 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 13:00:15.844746 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14967,"mutex_wait_us":24,"spinlock_wait_cycles":33792,"thread_start_us":15731,"threads_started":1}
I20260808 13:00:15.845741 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:15.819670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:45997 (local address 127.0.0.1:32974)
0808 13:00:15.835172 (+ 15502us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:15.835251 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:15.835515 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:15.836417 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:15.836424 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:15.836456 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:00:15.836875 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:15.836890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.839998 (+ 3108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.840003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:15.842599 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:15.842613 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.844779 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.844785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:15.844982 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:15.845015 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 13:00:15.845107 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 13:00:15.845248 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15127,"mutex_wait_us":88,"spinlock_wait_cycles":14080,"thread_start_us":15818,"threads_started":1}
I20260808 13:00:15.846657 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:15.819878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.24.167.3:45997)
0808 13:00:15.835164 (+ 15286us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:15.835174 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:15.835454 (+ 280us) server_negotiation.cc:407] Connection header received
0808 13:00:15.835862 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:15.835871 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:15.836150 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:15.836371 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:00:15.837110 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.839839 (+ 2729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.842802 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.844219 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.844360 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:15.844620 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:15.846058 (+ 1438us) server_negotiation.cc:299] Negotiation successful
0808 13:00:15.846250 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15099,"thread_start_us":15298,"threads_started":1}
I20260808 13:00:15.847381 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:15.820458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:43141 (local address 127.0.0.1:43200)
0808 13:00:15.838459 (+ 18001us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:15.838479 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:15.838595 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:15.839575 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:15.839580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:15.839608 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:00:15.839985 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:15.839995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.841976 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.841983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:15.844213 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:15.844244 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.846594 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.846600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:15.846800 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:15.846830 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 13:00:15.846941 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 13:00:15.847071 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17826,"thread_start_us":17666,"threads_started":1}
I20260808 13:00:15.847417 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:15.821839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.24.167.2:43141)
0808 13:00:15.836627 (+ 14788us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:15.836637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:15.838629 (+ 1992us) server_negotiation.cc:407] Connection header received
0808 13:00:15.838743 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:15.838752 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:15.839000 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:15.839194 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:00:15.840196 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.841789 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.844418 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:15.846058 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:15.846192 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:15.846444 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:15.847001 (+ 557us) server_negotiation.cc:299] Negotiation successful
0808 13:00:15.847166 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14615,"thread_start_us":14505,"threads_started":1}
I20260808 13:00:15.850677 15285 tablet_service.cc:1467] Processing CreateTablet for tablet b57cb01cf28145869165349a3fb505b3 (DEFAULT_TABLE table=test-table [id=46976cb1bef543bc9e63cb93af97a623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:00:15.850679 15073 tablet_service.cc:1467] Processing CreateTablet for tablet b57cb01cf28145869165349a3fb505b3 (DEFAULT_TABLE table=test-table [id=46976cb1bef543bc9e63cb93af97a623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:00:15.850680 15490 tablet_service.cc:1467] Processing CreateTablet for tablet b57cb01cf28145869165349a3fb505b3 (DEFAULT_TABLE table=test-table [id=46976cb1bef543bc9e63cb93af97a623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:00:15.852099 15285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57cb01cf28145869165349a3fb505b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:00:15.852178 15490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57cb01cf28145869165349a3fb505b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:00:15.852178 15073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57cb01cf28145869165349a3fb505b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:00:15.865447 15618 tablet_bootstrap.cc:492] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d: Bootstrap starting.
I20260808 13:00:15.868017 15619 tablet_bootstrap.cc:492] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830: Bootstrap starting.
I20260808 13:00:15.870172 15618 tablet_bootstrap.cc:654] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d: Neither blocks nor log segments found. Creating new log.
I20260808 13:00:15.871685 15618 log.cc:824] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:15.872380 15619 tablet_bootstrap.cc:654] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830: Neither blocks nor log segments found. Creating new log.
I20260808 13:00:15.873879 15619 log.cc:824] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:15.874747 15622 tablet_bootstrap.cc:492] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c: Bootstrap starting.
I20260808 13:00:15.875517 15618 tablet_bootstrap.cc:492] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d: No bootstrap required, opened a new log
I20260808 13:00:15.875875 15618 ts_tablet_manager.cc:1397] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260808 13:00:15.877646 15619 tablet_bootstrap.cc:492] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830: No bootstrap required, opened a new log
I20260808 13:00:15.878041 15619 ts_tablet_manager.cc:1397] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260808 13:00:15.878305 15622 tablet_bootstrap.cc:654] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c: Neither blocks nor log segments found. Creating new log.
I20260808 13:00:15.879693 15622 log.cc:824] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:15.882629 15622 tablet_bootstrap.cc:492] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c: No bootstrap required, opened a new log
I20260808 13:00:15.882912 15622 ts_tablet_manager.cc:1397] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260808 13:00:15.887272 15618 raft_consensus.cc:348] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:15.887809 15618 raft_consensus.cc:374] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:00:15.887972 15618 raft_consensus.cc:729] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a86eca8fd499441d8228eff286fbe56d, State: Initialized, Role: FOLLOWER
I20260808 13:00:15.888689 15618 consensus_queue.cc:260] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:15.889268 15619 raft_consensus.cc:348] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:15.889941 15619 raft_consensus.cc:374] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:00:15.890120 15619 raft_consensus.cc:729] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d651e120b9844df9f45c3fadd207830, State: Initialized, Role: FOLLOWER
I20260808 13:00:15.891093 15619 consensus_queue.cc:260] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:15.891404 15377 heartbeater.cc:503] Master 127.24.167.62:41391 was elected leader, sending a full tablet report...
I20260808 13:00:15.892217 15618 ts_tablet_manager.cc:1428] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260808 13:00:15.892933 15619 ts_tablet_manager.cc:1428] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260808 13:00:15.892799 15622 raft_consensus.cc:348] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:15.893314 15622 raft_consensus.cc:374] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:00:15.893455 15622 raft_consensus.cc:729] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a727ae0e680643f69a58203d7fd4a06c, State: Initialized, Role: FOLLOWER
I20260808 13:00:15.894088 15622 consensus_queue.cc:260] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:15.895291 15594 heartbeater.cc:503] Master 127.24.167.62:41391 was elected leader, sending a full tablet report...
I20260808 13:00:15.895779 15622 ts_tablet_manager.cc:1428] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260808 13:00:16.011945 15595 tablet.cc:2369] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 13:00:16.020043 15379 tablet.cc:2369] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 13:00:16.033074 15174 tablet.cc:2369] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:00:16.099617 15629 raft_consensus.cc:482] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 13:00:16.099932 15629 raft_consensus.cc:504] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:16.101711 15629 leader_election.cc:288] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a727ae0e680643f69a58203d7fd4a06c (127.24.167.3:45997), 8d651e120b9844df9f45c3fadd207830 (127.24.167.1:46425)
I20260808 13:00:16.109704 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:16.102222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:60001 (local address 127.24.167.3:45997)
0808 13:00:16.102673 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:16.102679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:16.102958 (+ 279us) server_negotiation.cc:407] Connection header received
0808 13:00:16.103211 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:16.103215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:16.103302 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:16.103423 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:00:16.103997 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.105061 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.107940 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.108733 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.108918 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:16.109001 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:16.109295 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 13:00:16.109423 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290}
I20260808 13:00:16.109719 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:16.102458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:44283 (local address 127.24.167.1:46425)
0808 13:00:16.102670 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:16.102677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:16.103333 (+ 656us) server_negotiation.cc:407] Connection header received
0808 13:00:16.103505 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:16.103511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:16.103595 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:16.103666 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:00:16.104327 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.105319 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.107878 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.108662 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.108869 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:16.108956 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:16.109304 (+ 348us) server_negotiation.cc:299] Negotiation successful
0808 13:00:16.109465 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 13:00:16.109874 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:16.102151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:46425 (local address 127.24.167.2:44283)
0808 13:00:16.103210 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:16.103225 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:16.103338 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:16.103716 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:16.103722 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:16.103745 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:00:16.104162 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:16.104171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.105438 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.105443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:16.107696 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:16.107714 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.108914 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.108921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:16.109100 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:16.109167 (+ 67us) client_negotiation.cc:769] Sending connection context
0808 13:00:16.109271 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 13:00:16.109415 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":44,"thread_start_us":717,"threads_started":1}
I20260808 13:00:16.109922 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:16.102067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:45997 (local address 127.24.167.2:60001)
0808 13:00:16.102845 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:16.102861 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:16.103056 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:16.103458 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:16.103462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:16.103490 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:00:16.103842 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:16.103859 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.105200 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.105204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:16.107789 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:16.107803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.108978 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.108985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:16.109145 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:16.109169 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 13:00:16.109268 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 13:00:16.109415 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":78,"thread_start_us":442,"threads_started":1}
I20260808 13:00:16.111202 15103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57cb01cf28145869165349a3fb505b3" candidate_uuid: "a86eca8fd499441d8228eff286fbe56d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d651e120b9844df9f45c3fadd207830" is_pre_election: true
I20260808 13:00:16.111358 15521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57cb01cf28145869165349a3fb505b3" candidate_uuid: "a86eca8fd499441d8228eff286fbe56d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a727ae0e680643f69a58203d7fd4a06c" is_pre_election: true
I20260808 13:00:16.111857 15103 raft_consensus.cc:2454] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a86eca8fd499441d8228eff286fbe56d in term 0.
I20260808 13:00:16.111996 15521 raft_consensus.cc:2454] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a86eca8fd499441d8228eff286fbe56d in term 0.
I20260808 13:00:16.112723 15215 leader_election.cc:302] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d651e120b9844df9f45c3fadd207830, a86eca8fd499441d8228eff286fbe56d; no voters:
I20260808 13:00:16.113173 15629 raft_consensus.cc:2781] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 13:00:16.113323 15629 raft_consensus.cc:482] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 13:00:16.113435 15629 raft_consensus.cc:3037] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:00:16.115830 15629 raft_consensus.cc:504] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:16.116477 15629 leader_election.cc:288] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [CANDIDATE]: Term 1 election: Requested vote from peers a727ae0e680643f69a58203d7fd4a06c (127.24.167.3:45997), 8d651e120b9844df9f45c3fadd207830 (127.24.167.1:46425)
I20260808 13:00:16.116822 15521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57cb01cf28145869165349a3fb505b3" candidate_uuid: "a86eca8fd499441d8228eff286fbe56d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a727ae0e680643f69a58203d7fd4a06c"
I20260808 13:00:16.117027 15521 raft_consensus.cc:3037] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:00:16.116953 15103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57cb01cf28145869165349a3fb505b3" candidate_uuid: "a86eca8fd499441d8228eff286fbe56d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d651e120b9844df9f45c3fadd207830"
I20260808 13:00:16.117199 15103 raft_consensus.cc:3037] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:00:16.119405 15521 raft_consensus.cc:2454] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a86eca8fd499441d8228eff286fbe56d in term 1.
I20260808 13:00:16.119588 15103 raft_consensus.cc:2454] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a86eca8fd499441d8228eff286fbe56d in term 1.
I20260808 13:00:16.119837 15216 leader_election.cc:302] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a727ae0e680643f69a58203d7fd4a06c, a86eca8fd499441d8228eff286fbe56d; no voters:
I20260808 13:00:16.120162 15629 raft_consensus.cc:2781] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:00:16.120573 15629 raft_consensus.cc:686] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [term 1 LEADER]: Becoming Leader. State: Replica: a86eca8fd499441d8228eff286fbe56d, State: Running, Role: LEADER
I20260808 13:00:16.121090 15629 consensus_queue.cc:237] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } } peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } }
I20260808 13:00:16.125523 14885 catalog_manager.cc:5495] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d reported cstate change: term changed from 0 to 1, leader changed from <none> to a86eca8fd499441d8228eff286fbe56d (127.24.167.2). New cstate: current_term: 1 leader_uuid: "a86eca8fd499441d8228eff286fbe56d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a86eca8fd499441d8228eff286fbe56d" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 43141 } health_report { overall_health: HEALTHY } } }
I20260808 13:00:16.145529 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:16.140410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.24.167.2:43141)
0808 13:00:16.140616 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:16.140620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:16.140638 (+ 18us) server_negotiation.cc:407] Connection header received
0808 13:00:16.140737 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:16.140741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:16.140817 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:16.140937 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:00:16.141267 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.142012 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.143497 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.143907 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.144022 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:16.144481 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:00:16.144987 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:00:16.145104 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:16.145239 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 13:00:16.145318 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 13:00:16.154109 15521 raft_consensus.cc:1261] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c [term 1 FOLLOWER]: Refusing update from remote peer a86eca8fd499441d8228eff286fbe56d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:00:16.154136 15103 raft_consensus.cc:1261] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830 [term 1 FOLLOWER]: Refusing update from remote peer a86eca8fd499441d8228eff286fbe56d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:00:16.155200 15636 consensus_queue.cc:1035] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a727ae0e680643f69a58203d7fd4a06c" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 45997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:00:16.155582 15629 consensus_queue.cc:1035] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d651e120b9844df9f45c3fadd207830" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:00:16.167682 15638 mvcc.cc:205] Tried to move back new op lower bound from 7316250690156863488 to 7316250690035646464. Current Snapshot: MvccSnapshot[applied={T|T < 7316250690156863488}]
I20260808 13:00:16.168358 15643 mvcc.cc:205] Tried to move back new op lower bound from 7316250690156863488 to 7316250690035646464. Current Snapshot: MvccSnapshot[applied={T|T < 7316250690156863488}]
I20260808 13:00:16.184789 14885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:00:16.185153 14885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:00:16.189384 14885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: Sending DeleteTablet for 3 replicas of tablet b57cb01cf28145869165349a3fb505b3
I20260808 13:00:16.190835 15490 tablet_service.cc:1514] Processing DeleteTablet for tablet b57cb01cf28145869165349a3fb505b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:00:16 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:32974
I20260808 13:00:16.190910 15073 tablet_service.cc:1514] Processing DeleteTablet for tablet b57cb01cf28145869165349a3fb505b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:00:16 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:51696
I20260808 13:00:16.191241 15285 tablet_service.cc:1514] Processing DeleteTablet for tablet b57cb01cf28145869165349a3fb505b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:00:16 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:43200
I20260808 13:00:16.192144 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 14945
I20260808 13:00:16.204034 15660 tablet_replica.cc:330] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d: stopping tablet replica
I20260808 13:00:16.204201 15659 tablet_replica.cc:330] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830: stopping tablet replica
I20260808 13:00:16.204820 15659 raft_consensus.cc:2229] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:00:16.205447 15659 raft_consensus.cc:2258] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:00:16.205487 15660 raft_consensus.cc:2229] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [term 1 LEADER]: Raft consensus shutting down.
I20260808 13:00:16.206342 15657 tablet_replica.cc:330] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c: stopping tablet replica
I20260808 13:00:16.206837 15657 raft_consensus.cc:2229] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:00:16.206943 15660 raft_consensus.cc:2258] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:00:16.207427 15657 raft_consensus.cc:2258] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:00:16.209275 15660 ts_tablet_manager.cc:1905] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:00:16.210798 15659 ts_tablet_manager.cc:1905] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:00:16.213840 15657 ts_tablet_manager.cc:1905] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:00:16.218633 15660 ts_tablet_manager.cc:1918] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:00:16.218724 15659 ts_tablet_manager.cc:1918] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:00:16.218935 15659 log.cc:1196] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/b57cb01cf28145869165349a3fb505b3
I20260808 13:00:16.218968 15660 log.cc:1196] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/b57cb01cf28145869165349a3fb505b3
I20260808 13:00:16.219703 15659 ts_tablet_manager.cc:1939] T b57cb01cf28145869165349a3fb505b3 P 8d651e120b9844df9f45c3fadd207830: Deleting consensus metadata
I20260808 13:00:16.219849 15660 ts_tablet_manager.cc:1939] T b57cb01cf28145869165349a3fb505b3 P a86eca8fd499441d8228eff286fbe56d: Deleting consensus metadata
I20260808 13:00:16.220007 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:16.193106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.24.167.1:46425)
0808 13:00:16.193350 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:16.193360 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:16.193389 (+ 29us) server_negotiation.cc:407] Connection header received
0808 13:00:16.193657 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:16.193664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:16.193758 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:16.193879 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:16.196181 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.197519 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.199960 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.200691 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.200849 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:16.206690 (+ 5841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:16.206708 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:16.206728 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:16.206774 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:16.212301 (+ 5527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:16.215021 (+ 2720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:16.215031 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:16.215036 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:16.215247 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:16.216210 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:16.216216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:16.216219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:16.216967 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:16.217331 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:16.219208 (+ 1877us) server_negotiation.cc:299] Negotiation successful
0808 13:00:16.219514 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 13:00:16.221793 15164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:16.221930 14862 catalog_manager.cc:4841] TS a86eca8fd499441d8228eff286fbe56d (127.24.167.2:43141): tablet b57cb01cf28145869165349a3fb505b3 (table test-table [id=46976cb1bef543bc9e63cb93af97a623]) successfully deleted
I20260808 13:00:16.222254 14863 catalog_manager.cc:4841] TS 8d651e120b9844df9f45c3fadd207830 (127.24.167.1:46425): tablet b57cb01cf28145869165349a3fb505b3 (table test-table [id=46976cb1bef543bc9e63cb93af97a623]) successfully deleted
I20260808 13:00:16.227345 15657 ts_tablet_manager.cc:1918] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:00:16.227744 15657 log.cc:1196] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/b57cb01cf28145869165349a3fb505b3
I20260808 13:00:16.228411 15657 ts_tablet_manager.cc:1939] T b57cb01cf28145869165349a3fb505b3 P a727ae0e680643f69a58203d7fd4a06c: Deleting consensus metadata
I20260808 13:00:16.230782 14865 catalog_manager.cc:4841] TS a727ae0e680643f69a58203d7fd4a06c (127.24.167.3:45997): tablet b57cb01cf28145869165349a3fb505b3 (table test-table [id=46976cb1bef543bc9e63cb93af97a623]) successfully deleted
I20260808 13:00:16.335935 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 14945
I20260808 13:00:16.360850 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 15183
I20260808 13:00:16.375945 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:16.361698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43224 (local address 127.24.167.2:43141)
0808 13:00:16.361880 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:16.361886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:16.361975 (+ 89us) server_negotiation.cc:407] Connection header received
0808 13:00:16.362202 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:16.362207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:16.362295 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:16.362454 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:16.363807 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.364679 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.366155 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.366751 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.366898 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:16.369680 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:16.369693 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:16.369709 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:16.369746 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:16.372891 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:16.373629 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:16.373636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:16.373639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:16.373785 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:16.374196 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:16.374202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:16.374205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:16.374686 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:16.375030 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:16.375346 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 13:00:16.375589 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 13:00:16.376690 15369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:16.491008 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 15183
I20260808 13:00:16.515952 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 15382
I20260808 13:00:16.530346 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:16.516795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32978 (local address 127.24.167.3:45997)
0808 13:00:16.516995 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:16.517000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:16.517019 (+ 19us) server_negotiation.cc:407] Connection header received
0808 13:00:16.517175 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:16.517179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:16.517245 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:16.517330 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:16.518689 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.519480 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.520829 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:16.521242 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:16.521333 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:16.523978 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:16.523992 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:16.524008 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:16.524052 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:16.527426 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:16.528205 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:16.528213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:16.528222 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:16.528405 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:16.528763 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:16.528767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:16.528769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:16.529179 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:16.529440 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:16.529745 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 13:00:16.529909 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 13:00:16.531272 15587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:16.622229 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 15382
I20260808 13:00:16.643833 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 14828
I20260808 13:00:16.645208 14934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:16.771930 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 14828
I20260808 13:00:16.803515 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:41391
--webserver_interface=127.24.167.62
--webserver_port=36235
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:41391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:00:17.172546 15667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:17.172863 15667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:17.173020 15667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:17.180568 15667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:00:17.180688 15667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:17.180778 15667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:17.180894 15667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:00:17.180959 15667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:00:17.191257 15667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:41391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=36235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:17.192634 15667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:17.193997 15667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:17.206133 15675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:17.206207 15677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:17.206454 15680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:17.207084 15667 server_base.cc:1034] running on GCE node
I20260808 13:00:17.207880 15667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:17.209329 15667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:17.210536 15667 hybrid_clock.cc:648] HybridClock initialized: now 1786194017210483 us; error 34 us; skew 500 ppm
I20260808 13:00:17.210923 15667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:17 dist-test-slave-tk9z krb5kdc[13898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194017, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:17.214267 15667 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:00:17.216039 15667 webserver.cc:460] Webserver started at http://127.24.167.62:36235/ using document root <none> and password file <none>
I20260808 13:00:17.216951 15667 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:17.217083 15667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:17.223517 15667 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 13:00:17.226899 15691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:17.228510 15667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 13:00:17.228684 15667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "0520edffaf2a4ee1ac7a31a6011424ae"
format_stamp: "Formatted at 2026-08-08 13:00:10 on dist-test-slave-tk9z"
server_key: "c16da6c002060beaa75c4da425a758fc"
server_key_iv: "cd0408edbd7004997948db7f8ee18468"
server_key_version: "encryptionkey@0"
I20260808 13:00:17.229763 15667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:17.248459 15667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:17.253592 15667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:17.253854 15667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:17.273851 15667 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:41391
I20260808 13:00:17.273891 15790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:41391 every 8 connection(s)
I20260808 13:00:17.275569 15667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:00:17.282877 15792 sys_catalog.cc:263] Verifying existing consensus state
I20260808 13:00:17.285662 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 15667
I20260808 13:00:17.286475 15792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: Bootstrap starting.
I20260808 13:00:17.286936 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:46425
--local_ip_for_outbound_sockets=127.24.167.1
--tserver_master_addrs=127.24.167.62:41391
--webserver_port=43833
--webserver_interface=127.24.167.1
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 13:00:17.307510 15792 log.cc:824] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:17.324697 15792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 13:00:17.325268 15792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae: Bootstrap complete.
I20260808 13:00:17.334954 15792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } }
I20260808 13:00:17.335929 15792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0520edffaf2a4ee1ac7a31a6011424ae, State: Initialized, Role: FOLLOWER
I20260808 13:00:17.336542 15792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } }
I20260808 13:00:17.336799 15792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:00:17.336944 15792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:00:17.337132 15792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 2 FOLLOWER]: Advancing to term 3
I20260808 13:00:17.341181 15792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } }
I20260808 13:00:17.341753 15792 leader_election.cc:302] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0520edffaf2a4ee1ac7a31a6011424ae; no voters:
I20260808 13:00:17.342523 15792 leader_election.cc:288] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 13:00:17.342623 15799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 3 FOLLOWER]: Leader election won for term 3
I20260808 13:00:17.344036 15799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [term 3 LEADER]: Becoming Leader. State: Replica: 0520edffaf2a4ee1ac7a31a6011424ae, State: Running, Role: LEADER
I20260808 13:00:17.344691 15799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } }
I20260808 13:00:17.345008 15792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:00:17.347688 15803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0520edffaf2a4ee1ac7a31a6011424ae. Latest consensus state: current_term: 3 leader_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } } }
I20260808 13:00:17.347697 15801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0520edffaf2a4ee1ac7a31a6011424ae" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 41391 } } }
I20260808 13:00:17.348285 15801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [sys.catalog]: This master's current role is: LEADER
I20260808 13:00:17.348291 15803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0520edffaf2a4ee1ac7a31a6011424ae [sys.catalog]: This master's current role is: LEADER
I20260808 13:00:17.359378 15809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:00:17.375356 15809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:00:17.376994 15809 catalog_manager.cc:1259] Loaded cluster ID: f4dd71da544742f093963bbbfe2c6e6b
I20260808 13:00:17.377207 15809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 13:00:17.379607 15809 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6dd547f9ad google::LogMessage::Fail() at ??:0
@ 0x7f6dd548288a google::LogMessage::SendToLog() at ??:0
@ 0x7f6dd547f4de google::LogMessage::Flush() at ??:0
@ 0x7f6dd5480a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6deaeb2512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6deaeaf976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6deaf37610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6deaf37439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6dd80cb345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6dd80ce91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6dd80ce799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6dd80a9eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6ddb3836db start_thread at ??:0
@ 0x7f6dd431961f clone at ??:0
W20260808 13:00:17.743491 15796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:17.743799 15796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:17.743971 15796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:17.751659 15796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:17.751842 15796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:17.752051 15796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 13:00:17.762334 15796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:46425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=43833
--tserver_master_addrs=127.24.167.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:17.763754 15796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:17.765146 15796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:17.779753 15860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:17.779788 15861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:17.780148 15864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:17.781226 15796 server_base.cc:1034] running on GCE node
I20260808 13:00:17.782095 15796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:17.783354 15796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:17.784646 15796 hybrid_clock.cc:648] HybridClock initialized: now 1786194017784579 us; error 55 us; skew 500 ppm
I20260808 13:00:17.785048 15796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:17 dist-test-slave-tk9z krb5kdc[13898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194017, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:17.788672 15796 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 13:00:17.790544 15796 webserver.cc:460] Webserver started at http://127.24.167.1:43833/ using document root <none> and password file <none>
I20260808 13:00:17.791319 15796 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:17.791487 15796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:17.797490 15796 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260808 13:00:17.800658 15878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:17.802429 15796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 13:00:17.802667 15796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "8d651e120b9844df9f45c3fadd207830"
format_stamp: "Formatted at 2026-08-08 13:00:11 on dist-test-slave-tk9z"
server_key: "d72ab587a8e5685b8eb04fffb1eefce7"
server_key_iv: "ff9c738dc4b01fbb84c9f6e1f250537a"
server_key_version: "encryptionkey@0"
I20260808 13:00:17.803859 15796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:17.827252 15796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:17.831990 15796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:17.832377 15796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:17.833470 15796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:00:17.835064 15796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:00:17.835203 15796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:17.835341 15796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:00:17.835436 15796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:17.872027 15796 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:46425
I20260808 13:00:17.872123 16058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:46425 every 8 connection(s)
I20260808 13:00:17.873558 15796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 13:00:17.878786 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 15796
I20260808 13:00:17.879243 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:17.876817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:47821)
0808 13:00:17.877648 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:17.877909 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":344,"threads_started":1}
I20260808 13:00:17.879580 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:43141
--local_ip_for_outbound_sockets=127.24.167.2
--tserver_master_addrs=127.24.167.62:41391
--webserver_port=40007
--webserver_interface=127.24.167.2
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:00:17.880064 16059 heartbeater.cc:644] Failed to heartbeat to 127.24.167.62:41391 (0 consecutive failures): Network error: Failed to ping master at 127.24.167.62:41391: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
I20260808 13:00:17.881147 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:17.880685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:60995)
0808 13:00:17.880884 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:17.880985 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 13:00:17.882144 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:17.881725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:48291)
0808 13:00:17.881893 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:17.881991 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 13:00:17.882447 16059 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 13:00:18.252818 16064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:18.253139 16064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:18.253333 16064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:18.260819 16064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:18.260936 16064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:18.261124 16064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 13:00:18.271340 16064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:43141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=40007
--tserver_master_addrs=127.24.167.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:18.272732 16064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:18.274083 16064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:18.287576 16072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:18.287618 16071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:18.287576 16075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:18.288641 16064 server_base.cc:1034] running on GCE node
I20260808 13:00:18.289292 16064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:18.290222 16064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:18.291515 16064 hybrid_clock.cc:648] HybridClock initialized: now 1786194018291445 us; error 77 us; skew 500 ppm
I20260808 13:00:18.291859 16064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:18 dist-test-slave-tk9z krb5kdc[13898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194018, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:18.295145 16064 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 13:00:18.296556 16064 webserver.cc:460] Webserver started at http://127.24.167.2:40007/ using document root <none> and password file <none>
I20260808 13:00:18.297252 16064 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:18.297392 16064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:18.302897 16064 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 13:00:18.305646 16083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:18.307067 16064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:00:18.307253 16064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "a86eca8fd499441d8228eff286fbe56d"
format_stamp: "Formatted at 2026-08-08 13:00:12 on dist-test-slave-tk9z"
server_key: "1436c608dc9950f505ecba7869b18d95"
server_key_iv: "bd4831dd40d11c60180b79ae552ac3a5"
server_key_version: "encryptionkey@0"
I20260808 13:00:18.308391 16064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:18.327136 16064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:18.331388 16064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:18.331698 16064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:18.332806 16064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:00:18.334134 16064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:00:18.334275 16064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:18.334430 16064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:00:18.334539 16064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260808 13:00:18.367544 16064 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:43141
I20260808 13:00:18.367604 16266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:43141 every 8 connection(s)
I20260808 13:00:18.369069 16064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 13:00:18.370512 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 16064
I20260808 13:00:18.371280 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:45997
--local_ip_for_outbound_sockets=127.24.167.3
--tserver_master_addrs=127.24.167.62:41391
--webserver_port=37705
--webserver_interface=127.24.167.3
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 13:00:18.374857 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:18.372481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:51719)
0808 13:00:18.373362 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:18.373645 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":316,"threads_started":1}
W20260808 13:00:18.375656 16267 heartbeater.cc:644] Failed to heartbeat to 127.24.167.62:41391 (0 consecutive failures): Network error: Failed to ping master at 127.24.167.62:41391: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
I20260808 13:00:18.376655 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:18.376216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:58129)
0808 13:00:18.376406 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:18.376502 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 13:00:18.377656 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:18.377223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:36255)
0808 13:00:18.377396 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:18.377500 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 13:00:18.377913 16267 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 13:00:18.716180 16270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:18.716473 16270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:18.716670 16270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:18.723949 16270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:18.724061 16270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:18.724334 16270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 13:00:18.734001 16270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:45997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=37705
--tserver_master_addrs=127.24.167.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:18.735324 16270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:18.736654 16270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:18.750734 16278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:18.750698 16289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:18.750991 16279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:18.751654 16270 server_base.cc:1034] running on GCE node
I20260808 13:00:18.752390 16270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:18.753432 16270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:18.754647 16270 hybrid_clock.cc:648] HybridClock initialized: now 1786194018754599 us; error 33 us; skew 500 ppm
I20260808 13:00:18.755009 16270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:18 dist-test-slave-tk9z krb5kdc[13898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194018, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:18.758656 16270 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 13:00:18.760170 16270 webserver.cc:460] Webserver started at http://127.24.167.3:37705/ using document root <none> and password file <none>
I20260808 13:00:18.760896 16270 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:18.761020 16270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:18.766988 16270 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 13:00:18.769954 16298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:18.771446 16270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:00:18.771651 16270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "a727ae0e680643f69a58203d7fd4a06c"
format_stamp: "Formatted at 2026-08-08 13:00:12 on dist-test-slave-tk9z"
server_key: "8a98fc85d8748f1019f7d19274b0d278"
server_key_iv: "ca839cbbd1c7506dce54f48fabedbba7"
server_key_version: "encryptionkey@0"
I20260808 13:00:18.772778 16270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:18.804490 16270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:18.809661 16270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:18.809983 16270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:18.811249 16270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:00:18.812907 16270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:00:18.813045 16270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:18.813201 16270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:00:18.813313 16270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:18.850970 16270 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:45997
I20260808 13:00:18.851035 16476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:45997 every 8 connection(s)
I20260808 13:00:18.852589 16270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 13:00:18.857652 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:18.855767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:55567)
0808 13:00:18.856536 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:18.856799 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":291,"threads_started":1}
W20260808 13:00:18.858485 16477 heartbeater.cc:644] Failed to heartbeat to 127.24.167.62:41391 (0 consecutive failures): Network error: Failed to ping master at 127.24.167.62:41391: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
I20260808 13:00:18.859285 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:18.858958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:47447)
0808 13:00:18.859113 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:18.859188 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 13:00:18.860134 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:18.859797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:36429)
0808 13:00:18.859937 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:18.860022 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 13:00:18.860414 16477 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 13:00:18.860831 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 16270
I20260808 13:00:18.883945 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:18.883103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:47463)
0808 13:00:18.883584 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:18.883689 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":221,"threads_started":1}
I20260808 13:00:19.379410 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:19.378512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:41327)
0808 13:00:19.379061 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:19.379183 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":294,"threads_started":1}
I20260808 13:00:19.862004 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:19.861077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:48167)
0808 13:00:19.861636 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:19.861752 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":300,"threads_started":1}
I20260808 13:00:19.885699 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:19.884917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:35855)
0808 13:00:19.885415 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:19.885510 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":237,"threads_started":1}
I20260808 13:00:20.381114 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:20.380308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:48613)
0808 13:00:20.380797 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:20.380909 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":240,"threads_started":1}
I20260808 13:00:20.864151 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:20.863217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:57505)
0808 13:00:20.863891 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:20.863979 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":422,"threads_started":1}
I20260808 13:00:20.888072 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:20.886808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:52093)
0808 13:00:20.887763 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:20.887873 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":701,"threads_started":1}
I20260808 13:00:21.383038 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:21.382038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:43079)
0808 13:00:21.382692 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:21.382788 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":383,"threads_started":1}
I20260808 13:00:21.865942 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:21.865176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:37931)
0808 13:00:21.865648 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:21.865736 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":249,"threads_started":1}
I20260808 13:00:21.890043 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:21.889146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:58771)
0808 13:00:21.889753 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:21.889846 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":358,"threads_started":1}
I20260808 13:00:22.385066 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:22.384058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:51659)
0808 13:00:22.384704 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:22.384804 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":387,"threads_started":1}
I20260808 13:00:22.867837 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:22.867068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:40727)
0808 13:00:22.867555 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:22.867649 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":247,"threads_started":1}
I20260808 13:00:22.892073 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:22.891201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:50945)
0808 13:00:22.891784 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:22.891866 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":347,"threads_started":1}
I20260808 13:00:23.392412 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:23.391606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:40291)
0808 13:00:23.392103 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:23.392189 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":252,"threads_started":1}
I20260808 13:00:23.869796 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:23.868992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:52761)
0808 13:00:23.869534 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:23.869624 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":277,"threads_started":1}
I20260808 13:00:23.893841 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:23.893042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:48785)
0808 13:00:23.893571 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:23.893670 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":284,"threads_started":1}
I20260808 13:00:24.394649 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:24.393602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:47609)
0808 13:00:24.394258 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:24.394426 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":359,"threads_started":1}
I20260808 13:00:24.871591 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:24.870870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:34389)
0808 13:00:24.871321 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:24.871422 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":230,"threads_started":1}
I20260808 13:00:24.895934 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:24.895061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:50133)
0808 13:00:24.895617 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:24.895711 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":237,"threads_started":1}
I20260808 13:00:25.396642 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:25.395691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:39517)
0808 13:00:25.396224 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:25.396355 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":229,"threads_started":1}
I20260808 13:00:25.873574 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:25.872630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:47835)
0808 13:00:25.873218 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:25.873313 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":282,"threads_started":1}
I20260808 13:00:25.898043 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:25.897235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:53779)
0808 13:00:25.897748 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:25.897832 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":278,"threads_started":1}
I20260808 13:00:26.398555 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:26.397697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:54523)
0808 13:00:26.398162 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:26.398253 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":236,"threads_started":1}
I20260808 13:00:26.875567 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:26.874697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:56867)
0808 13:00:26.875279 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:26.875396 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":311,"threads_started":1}
I20260808 13:00:26.899734 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:26.899011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:54795)
0808 13:00:26.899459 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:26.899548 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":214,"threads_started":1}
I20260808 13:00:27.400456 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:27.399563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:45973)
0808 13:00:27.400164 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:27.400261 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":298,"threads_started":1}
I20260808 13:00:27.877430 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:27.876560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:46109)
0808 13:00:27.877154 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:27.877246 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":286,"threads_started":1}
I20260808 13:00:27.901563 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:27.900815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:33683)
0808 13:00:27.901300 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:27.901388 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":271,"threads_started":1}
I20260808 13:00:28.402421 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:28.401471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:56335)
0808 13:00:28.401999 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:28.402131 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":246,"threads_started":1}
I20260808 13:00:28.879464 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:28.878607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:39289)
0808 13:00:28.879131 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:28.879248 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":289,"threads_started":1}
I20260808 13:00:28.903575 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:28.902625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:46077)
0808 13:00:28.903215 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:28.903356 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":285,"threads_started":1}
I20260808 13:00:29.404366 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:29.403433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:44983)
0808 13:00:29.404039 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:29.404167 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":263,"threads_started":1}
I20260808 13:00:29.881319 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:29.880489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:52065)
0808 13:00:29.881025 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:29.881136 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":252,"threads_started":1}
I20260808 13:00:29.905906 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:29.904967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:48103)
0808 13:00:29.905571 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:29.905701 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":298,"threads_started":1}
I20260808 13:00:30.406294 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:30.405372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:52223)
0808 13:00:30.406010 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:30.406105 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":247,"threads_started":1}
I20260808 13:00:30.883209 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:30.882459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:47251)
0808 13:00:30.882949 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:30.883043 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":232,"threads_started":1}
I20260808 13:00:30.908485 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:30.907187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:54245)
0808 13:00:30.907981 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:30.908170 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":319,"threads_started":1}
I20260808 13:00:31.408182 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:31.407332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:38225)
0808 13:00:31.407797 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:31.407972 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":212,"threads_started":1}
I20260808 13:00:31.885100 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:31.884315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:50785)
0808 13:00:31.884801 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:31.884907 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":232,"threads_started":1}
I20260808 13:00:31.910211 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:31.909483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:53317)
0808 13:00:31.909930 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:31.910021 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":218,"threads_started":1}
I20260808 13:00:32.409984 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:32.409207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:45941)
0808 13:00:32.409703 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:32.409795 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":279,"threads_started":1}
I20260808 13:00:32.886997 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:32.886075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:38929)
0808 13:00:32.886686 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:32.886777 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":395,"threads_started":1}
I20260808 13:00:32.912282 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:32.911402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.1:38313)
0808 13:00:32.911935 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:32.912054 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":297,"threads_started":1}
I20260808 13:00:33.411917 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:33.411164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.2:41109)
0808 13:00:33.411642 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:33.411733 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":262,"threads_started":1}
I20260808 13:00:33.863049 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 15796
I20260808 13:00:33.883112 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:33.864398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.24.167.1:46425)
0808 13:00:33.865169 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:33.865185 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:33.865226 (+ 41us) server_negotiation.cc:407] Connection header received
0808 13:00:33.866981 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:33.866993 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:33.867431 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:33.867736 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:33.869405 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:33.870799 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:33.872244 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:33.872932 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:33.873092 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:33.876220 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:33.876228 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:33.876245 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:33.876291 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:33.879633 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:33.880423 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:33.880430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:33.880437 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:33.880619 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:33.881006 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:33.881009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:33.881011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:33.881571 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:33.881875 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:33.882355 (+ 480us) server_negotiation.cc:299] Negotiation successful
0808 13:00:33.882791 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":340,"threads_started":1}
I20260808 13:00:33.884593 16053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:33.889375 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:33.888321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:41391 (local address 127.24.167.3:57397)
0808 13:00:33.888873 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:33.889042 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":303,"threads_started":1}
I20260808 13:00:33.989774 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 15796
I20260808 13:00:34.021135 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 16064
I20260808 13:00:34.040567 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:34.022242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42712 (local address 127.24.167.2:43141)
0808 13:00:34.022867 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:34.022882 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:34.022921 (+ 39us) server_negotiation.cc:407] Connection header received
0808 13:00:34.024588 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:34.024597 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:34.025032 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:34.025391 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:34.027221 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:34.028523 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:34.029804 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:34.030603 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:34.030822 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:34.033662 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:34.033670 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:34.033685 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:34.033725 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:34.036817 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:34.037553 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:34.037560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:34.037567 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:34.037753 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:34.038212 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:34.038218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:34.038221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:34.038950 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:34.039292 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:34.039807 (+ 515us) server_negotiation.cc:299] Negotiation successful
0808 13:00:34.040275 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":332,"threads_started":1}
I20260808 13:00:34.042068 16259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:34.129488 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 16064
I20260808 13:00:34.156625 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 16270
I20260808 13:00:34.175690 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:34.157736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.24.167.3:45997)
0808 13:00:34.158467 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:34.158483 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:34.158529 (+ 46us) server_negotiation.cc:407] Connection header received
0808 13:00:34.160290 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:34.160301 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:34.160822 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:34.161169 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:34.162926 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:34.164209 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:34.165524 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:34.166186 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:34.166387 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:34.169138 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:34.169145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:34.169160 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:34.169204 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:34.172184 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:34.172948 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:34.172963 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:34.172970 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:34.173140 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:34.173537 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:34.173543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:34.173546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:34.174094 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:34.174449 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:34.174924 (+ 475us) server_negotiation.cc:299] Negotiation successful
0808 13:00:34.175380 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":330,"threads_started":1}
I20260808 13:00:34.177211 16472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:34.264400 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 16270
2026-08-08T13:00:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25389 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:00:34 dist-test-slave-tk9z krb5kdc[16570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:00:34 dist-test-slave-tk9z krb5kdc[16570](info): set up 2 sockets
Aug 08 13:00:34 dist-test-slave-tk9z krb5kdc[16570](info): commencing operation
krb5kdc: starting...
W20260808 13:00:35.676885 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.364s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:00:35 dist-test-slave-tk9z krb5kdc[16570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:00:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:00:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 13:00:35.978526 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:43679
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:43679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:00:36.341192 16591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:36.341501 16591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:36.341671 16591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:36.349188 16591 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 13:00:36.349298 16591 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 13:00:36.349390 16591 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 13:00:36.349505 16591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:00:36.349601 16591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:36.349726 16591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:36.349809 16591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:00:36.349895 16591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:00:36.360483 16591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:43679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:43679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:36.361881 16591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:36.363297 16591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:36.376116 16603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:36.376201 16604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:36.376806 16591 server_base.cc:1034] running on GCE node
W20260808 13:00:36.376830 16606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:36.377822 16591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:36.379323 16591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:36.380551 16591 hybrid_clock.cc:648] HybridClock initialized: now 1786194036380489 us; error 30 us; skew 500 ppm
I20260808 13:00:36.380904 16591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:36 dist-test-slave-tk9z krb5kdc[16570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194036, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:36.387391 16591 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:00:36.389109 16591 webserver.cc:460] Webserver started at http://127.24.167.62:38055/ using document root <none> and password file <none>
I20260808 13:00:36.389870 16591 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:36.390013 16591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:36.390431 16591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:00:36.393651 16591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "d7335399a36e424da32d6a3b5c8c32a4"
format_stamp: "Formatted at 2026-08-08 13:00:36 on dist-test-slave-tk9z"
server_key: "d7f1c3616429fc452efd402e0ece60a7"
server_key_iv: "76ecc67eb9f5b27fac1edc61f9466a31"
server_key_version: "encryptionkey@0"
I20260808 13:00:36.394656 16591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "d7335399a36e424da32d6a3b5c8c32a4"
format_stamp: "Formatted at 2026-08-08 13:00:36 on dist-test-slave-tk9z"
server_key: "d7f1c3616429fc452efd402e0ece60a7"
server_key_iv: "76ecc67eb9f5b27fac1edc61f9466a31"
server_key_version: "encryptionkey@0"
I20260808 13:00:36.400344 16591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 13:00:36.404050 16617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:36.405433 16591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 13:00:36.405646 16591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "d7335399a36e424da32d6a3b5c8c32a4"
format_stamp: "Formatted at 2026-08-08 13:00:36 on dist-test-slave-tk9z"
server_key: "d7f1c3616429fc452efd402e0ece60a7"
server_key_iv: "76ecc67eb9f5b27fac1edc61f9466a31"
server_key_version: "encryptionkey@0"
I20260808 13:00:36.405895 16591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:36.427726 16591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:36.432241 16591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:36.432560 16591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:36.452644 16591 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:43679
I20260808 13:00:36.452669 16693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:43679 every 8 connection(s)
I20260808 13:00:36.454087 16591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:00:36.458153 16694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:00:36.459323 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 16591
I20260808 13:00:36.459570 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:00:36.460268 25244 external_mini_cluster.cc:1428] Setting key fddbe94b4e03d66f04d76a0424e44a8d
I20260808 13:00:36.472985 16694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Bootstrap starting.
Aug 08 13:00:36 dist-test-slave-tk9z krb5kdc[16570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:00:36.476233 16694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Neither blocks nor log segments found. Creating new log.
I20260808 13:00:36.477667 16694 log.cc:824] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:36.481382 16694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: No bootstrap required, opened a new log
I20260808 13:00:36.486068 16699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:36.462410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.24.167.62:43679)
0808 13:00:36.463321 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:36.463341 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:36.463395 (+ 54us) server_negotiation.cc:407] Connection header received
0808 13:00:36.465127 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:36.465157 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:36.465660 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:36.466127 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:36.467648 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:36.469710 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:36.471120 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:36.472165 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:36.472339 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:36.476796 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:36.476809 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:36.476828 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:36.476882 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:36.481119 (+ 4237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:36.481918 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:36.481926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:36.481934 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:36.482168 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:36.482627 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:36.482632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:36.482635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:36.483384 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:36.483710 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:36.484102 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 13:00:36.484570 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":380,"threads_started":1}
I20260808 13:00:36.491369 16694 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } }
I20260808 13:00:36.491734 16694 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:00:36.491811 16694 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7335399a36e424da32d6a3b5c8c32a4, State: Initialized, Role: FOLLOWER
I20260808 13:00:36.492354 16694 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } }
I20260808 13:00:36.492592 16694 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:00:36.492690 16694 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:00:36.492821 16694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:00:36.495271 16694 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } }
I20260808 13:00:36.495875 16694 leader_election.cc:302] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7335399a36e424da32d6a3b5c8c32a4; no voters:
I20260808 13:00:36.496469 16694 leader_election.cc:288] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:00:36.496608 16702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:00:36.497421 16702 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 1 LEADER]: Becoming Leader. State: Replica: d7335399a36e424da32d6a3b5c8c32a4, State: Running, Role: LEADER
I20260808 13:00:36.498068 16702 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } }
I20260808 13:00:36.498718 16694 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:00:36.500876 16703 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7335399a36e424da32d6a3b5c8c32a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } } }
I20260808 13:00:36.501289 16703 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [sys.catalog]: This master's current role is: LEADER
I20260808 13:00:36.501763 16705 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7335399a36e424da32d6a3b5c8c32a4. Latest consensus state: current_term: 1 leader_uuid: "d7335399a36e424da32d6a3b5c8c32a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } } }
I20260808 13:00:36.502087 16705 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [sys.catalog]: This master's current role is: LEADER
I20260808 13:00:36.503839 16713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:00:36.509189 16713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:00:36.518443 16713 catalog_manager.cc:1347] Generated new cluster ID: 80bbd6c7076446359f9faf247c4ce482
I20260808 13:00:36.518571 16713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:00:36.543898 16713 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:00:36.545449 16713 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:00:36.563586 16713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Generated new TSK 0
I20260808 13:00:36.564742 16713 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 13:00:36.640136 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:43679
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:00:36.999224 16745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:36.999560 16745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:36.999691 16745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:37.007647 16745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:37.007781 16745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:37.007905 16745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 13:00:37.018249 16745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:43679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:37.019726 16745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:37.021096 16745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:37.035107 16758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:37.035151 16753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:37.035341 16755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:37.036213 16745 server_base.cc:1034] running on GCE node
I20260808 13:00:37.036862 16745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:37.037833 16745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:37.039081 16745 hybrid_clock.cc:648] HybridClock initialized: now 1786194037039035 us; error 39 us; skew 500 ppm
I20260808 13:00:37.039469 16745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:37 dist-test-slave-tk9z krb5kdc[16570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194037, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:37.042872 16745 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 13:00:37.044443 16745 webserver.cc:460] Webserver started at http://127.24.167.1:45813/ using document root <none> and password file <none>
I20260808 13:00:37.045118 16745 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:37.045224 16745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:37.045581 16745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:00:37.048789 16745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "36c22d98dda44b5ebd1ac35bd84dbd33"
format_stamp: "Formatted at 2026-08-08 13:00:37 on dist-test-slave-tk9z"
server_key: "43c0cd2965b824e8d3055229ea7f594c"
server_key_iv: "0bc28cd3f579af634199aa24a844ffa6"
server_key_version: "encryptionkey@0"
I20260808 13:00:37.049597 16745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "36c22d98dda44b5ebd1ac35bd84dbd33"
format_stamp: "Formatted at 2026-08-08 13:00:37 on dist-test-slave-tk9z"
server_key: "43c0cd2965b824e8d3055229ea7f594c"
server_key_iv: "0bc28cd3f579af634199aa24a844ffa6"
server_key_version: "encryptionkey@0"
I20260808 13:00:37.055035 16745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 13:00:37.059351 16775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:37.060879 16745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260808 13:00:37.061045 16745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "36c22d98dda44b5ebd1ac35bd84dbd33"
format_stamp: "Formatted at 2026-08-08 13:00:37 on dist-test-slave-tk9z"
server_key: "43c0cd2965b824e8d3055229ea7f594c"
server_key_iv: "0bc28cd3f579af634199aa24a844ffa6"
server_key_version: "encryptionkey@0"
I20260808 13:00:37.061241 16745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:37.082080 16745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:37.086458 16745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:37.086730 16745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:37.087754 16745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:00:37.089200 16745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:00:37.089284 16745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:37.089377 16745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:00:37.089421 16745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:37.126691 16745 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:45195
I20260808 13:00:37.126744 16961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:45195 every 8 connection(s)
I20260808 13:00:37.128094 16745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 13:00:37.130470 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 16745
I20260808 13:00:37.130702 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 13:00:37.131682 25244 external_mini_cluster.cc:1428] Setting key 69eae7034f920ec2f92f7803c0557366
Aug 08 13:00:37 dist-test-slave-tk9z krb5kdc[16570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194037, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:00:37.152750 16967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:37.131190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:53309 (local address 127.24.167.62:43679)
0808 13:00:37.131763 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:37.131769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:37.132919 (+ 1150us) server_negotiation.cc:407] Connection header received
0808 13:00:37.135537 (+ 2618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:37.135542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:37.135625 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:37.135723 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:37.138054 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:37.138909 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:37.141491 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:37.141924 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:37.142044 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:37.146587 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:37.146593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:37.146596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:37.146633 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:37.149469 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:37.150277 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:37.150284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:37.150288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:37.150483 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:37.150945 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:37.150951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:37.150954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:37.151232 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:37.151399 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:37.152127 (+ 728us) server_negotiation.cc:299] Negotiation successful
0808 13:00:37.152367 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":283,"threads_started":1}
I20260808 13:00:37.153825 16968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:37.131803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:53309)
0808 13:00:37.132715 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:37.132779 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:37.135197 (+ 2418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:37.136043 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:37.136059 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:37.136942 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:37.137824 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:37.137853 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:37.139081 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:37.139089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:37.141312 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:37.141332 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:37.142186 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:37.142196 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:37.142586 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:37.143432 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:37.143474 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:37.146287 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:37.149645 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:37.149655 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:37.149673 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:37.150133 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:37.150602 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:37.150608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:37.150611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:37.150827 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:37.151372 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:37.151380 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:37.151747 (+ 367us) client_negotiation.cc:769] Sending connection context
0808 13:00:37.152031 (+ 284us) client_negotiation.cc:240] Negotiation successful
0808 13:00:37.152603 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":327,"threads_started":1}
I20260808 13:00:37.155664 16962 heartbeater.cc:347] Connected to a master server at 127.24.167.62:43679
I20260808 13:00:37.156014 16962 heartbeater.cc:464] Registering TS with master...
I20260808 13:00:37.156803 16962 heartbeater.cc:511] Master 127.24.167.62:43679 requested a full tablet report, sending...
I20260808 13:00:37.159505 16639 ts_manager.cc:194] Registered new tserver with Master: 36c22d98dda44b5ebd1ac35bd84dbd33 (127.24.167.1:45195)
I20260808 13:00:37.161814 16639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:53309
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 13:00:37.199530 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:43679
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:00:37.555797 16976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:37.556285 16976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:37.556489 16976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:37.564043 16976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:37.564216 16976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:37.564414 16976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 13:00:37.574604 16976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:43679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:37.576058 16976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:37.577450 16976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:37.591887 16984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:37.592041 16990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:37.592320 16986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:37.593185 16976 server_base.cc:1034] running on GCE node
I20260808 13:00:37.593930 16976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:37.595038 16976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:37.596261 16976 hybrid_clock.cc:648] HybridClock initialized: now 1786194037596191 us; error 66 us; skew 500 ppm
I20260808 13:00:37.596634 16976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:37 dist-test-slave-tk9z krb5kdc[16570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194037, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:37.600239 16976 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 13:00:37.601748 16976 webserver.cc:460] Webserver started at http://127.24.167.2:34853/ using document root <none> and password file <none>
I20260808 13:00:37.602528 16976 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:37.602679 16976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:37.602999 16976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:00:37.606094 16976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "fbed5731bdde46f281e4c55af33c61fa"
format_stamp: "Formatted at 2026-08-08 13:00:37 on dist-test-slave-tk9z"
server_key: "04ed2b600a808f6d2dcdfd2b8d3f736f"
server_key_iv: "6cebe99f537bcea6bc68a0daf5f6932e"
server_key_version: "encryptionkey@0"
I20260808 13:00:37.606928 16976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "fbed5731bdde46f281e4c55af33c61fa"
format_stamp: "Formatted at 2026-08-08 13:00:37 on dist-test-slave-tk9z"
server_key: "04ed2b600a808f6d2dcdfd2b8d3f736f"
server_key_iv: "6cebe99f537bcea6bc68a0daf5f6932e"
server_key_version: "encryptionkey@0"
I20260808 13:00:37.612236 16976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 13:00:37.615779 17001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:37.617174 16976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 13:00:37.617391 16976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "fbed5731bdde46f281e4c55af33c61fa"
format_stamp: "Formatted at 2026-08-08 13:00:37 on dist-test-slave-tk9z"
server_key: "04ed2b600a808f6d2dcdfd2b8d3f736f"
server_key_iv: "6cebe99f537bcea6bc68a0daf5f6932e"
server_key_version: "encryptionkey@0"
I20260808 13:00:37.617625 16976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:37.629276 16976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:37.633826 16976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:37.634145 16976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:37.635161 16976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:00:37.636552 16976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:00:37.636660 16976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:37.636818 16976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:00:37.636893 16976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:37.675325 16976 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:36913
I20260808 13:00:37.675390 17169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:36913 every 8 connection(s)
I20260808 13:00:37.676733 16976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 13:00:37.681249 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 16976
I20260808 13:00:37.681506 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 13:00:37.682018 25244 external_mini_cluster.cc:1428] Setting key 2ec7014a20aaa54707e7d701a7155945
Aug 08 13:00:37 dist-test-slave-tk9z krb5kdc[16570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194037, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:00:37.698890 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:37.679805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:38793 (local address 127.24.167.62:43679)
0808 13:00:37.680284 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:37.680289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:37.681337 (+ 1048us) server_negotiation.cc:407] Connection header received
0808 13:00:37.683651 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:37.683656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:37.683724 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:37.683841 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:37.685503 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:37.686249 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:37.687932 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:37.688366 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:37.688501 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:37.692410 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:37.692415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:37.692418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:37.692453 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:37.695381 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:37.696152 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:37.696161 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:37.696166 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:37.696354 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:37.696806 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:37.696814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:37.696820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:37.697232 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:37.697405 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:37.698099 (+ 694us) server_negotiation.cc:299] Negotiation successful
0808 13:00:37.698398 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":259,"threads_started":1}
I20260808 13:00:37.699774 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:37.680359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:38793)
0808 13:00:37.681153 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:37.681194 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:37.683233 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:37.684146 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:37.684155 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:37.684708 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:37.685320 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:37.685339 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:37.686464 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:37.686468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:37.687768 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:37.687787 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:37.688617 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:37.688624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:37.688864 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:37.689575 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:37.689598 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:37.692151 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:37.695549 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:37.695556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:37.695570 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:37.696005 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:37.696492 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:37.696495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:37.696497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:37.696684 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:37.697365 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:37.697370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:37.697714 (+ 344us) client_negotiation.cc:769] Sending connection context
0808 13:00:37.697937 (+ 223us) client_negotiation.cc:240] Negotiation successful
0808 13:00:37.698330 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":331,"threads_started":1}
I20260808 13:00:37.701274 17170 heartbeater.cc:347] Connected to a master server at 127.24.167.62:43679
I20260808 13:00:37.701612 17170 heartbeater.cc:464] Registering TS with master...
I20260808 13:00:37.702337 17170 heartbeater.cc:511] Master 127.24.167.62:43679 requested a full tablet report, sending...
I20260808 13:00:37.704011 16639 ts_manager.cc:194] Registered new tserver with Master: fbed5731bdde46f281e4c55af33c61fa (127.24.167.2:36913)
I20260808 13:00:37.705354 16639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:38793
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 13:00:37.749907 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:43679
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:00:38.104857 17183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:38.105230 17183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:38.105346 17183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:38.113206 17183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:38.113345 17183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:38.113473 17183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 13:00:38.124835 17183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:43679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:38.126468 17183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:38.128242 17183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:38.143371 17196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:38.143487 17189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:38.143682 17188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:38.144500 17183 server_base.cc:1034] running on GCE node
I20260808 13:00:38.145195 17183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:38.146153 17183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:38.147390 17183 hybrid_clock.cc:648] HybridClock initialized: now 1786194038147323 us; error 53 us; skew 500 ppm
I20260808 13:00:38.147735 17183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:38 dist-test-slave-tk9z krb5kdc[16570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194038, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:38.151111 17183 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 13:00:38.152704 17183 webserver.cc:460] Webserver started at http://127.24.167.3:44311/ using document root <none> and password file <none>
I20260808 13:00:38.153430 17183 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:38.153587 17183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:38.153913 17183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:00:38.157001 17183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "24db0abfd3c545dd8aec45a422da099e"
format_stamp: "Formatted at 2026-08-08 13:00:38 on dist-test-slave-tk9z"
server_key: "8b3288ea8ec68d1d318ed0e6fc6f1b47"
server_key_iv: "cf026565e609df9f308435af70268ab0"
server_key_version: "encryptionkey@0"
I20260808 13:00:38.157807 17183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "24db0abfd3c545dd8aec45a422da099e"
format_stamp: "Formatted at 2026-08-08 13:00:38 on dist-test-slave-tk9z"
server_key: "8b3288ea8ec68d1d318ed0e6fc6f1b47"
server_key_iv: "cf026565e609df9f308435af70268ab0"
server_key_version: "encryptionkey@0"
I20260808 13:00:38.163177 17183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 13:00:38.165546 16962 heartbeater.cc:503] Master 127.24.167.62:43679 was elected leader, sending a full tablet report...
I20260808 13:00:38.166805 17205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:38.168335 17183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 13:00:38.168517 17183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "24db0abfd3c545dd8aec45a422da099e"
format_stamp: "Formatted at 2026-08-08 13:00:38 on dist-test-slave-tk9z"
server_key: "8b3288ea8ec68d1d318ed0e6fc6f1b47"
server_key_iv: "cf026565e609df9f308435af70268ab0"
server_key_version: "encryptionkey@0"
I20260808 13:00:38.168758 17183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:38.222301 17183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:38.227094 17183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:38.227427 17183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:38.228545 17183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:00:38.229984 17183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:00:38.230070 17183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:38.230248 17183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:00:38.230321 17183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:38.267158 17183 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:40393
I20260808 13:00:38.267208 17381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:40393 every 8 connection(s)
I20260808 13:00:38.268528 17183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 13:00:38.270998 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 17183
I20260808 13:00:38.271209 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 13:00:38.271853 25244 external_mini_cluster.cc:1428] Setting key a118a2c0a4eca7371ba4faccd645316d
Aug 08 13:00:38 dist-test-slave-tk9z krb5kdc[16570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194038, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:00:38.290234 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.271262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:35903 (local address 127.24.167.62:43679)
0808 13:00:38.271760 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:38.271768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:38.273004 (+ 1236us) server_negotiation.cc:407] Connection header received
0808 13:00:38.275548 (+ 2544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:38.275553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:38.275633 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:38.275726 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:38.277276 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.278134 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.279754 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.280170 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.280280 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.284010 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:38.284015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:38.284019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:38.284053 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:38.287027 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:38.288020 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:38.288025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:38.288029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:38.288178 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:38.288625 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:38.288629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:38.288631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:38.288913 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:38.289042 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:38.289771 (+ 729us) server_negotiation.cc:299] Negotiation successful
0808 13:00:38.289961 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":233,"threads_started":1}
I20260808 13:00:38.291307 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.271857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:35903)
0808 13:00:38.272793 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:38.272844 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:38.275227 (+ 2383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:38.275963 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:38.275970 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:38.276492 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:00:38.277086 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:38.277109 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.278291 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.278295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:38.279603 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:38.279615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.280375 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.280380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:38.280586 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.281264 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:00:38.281286 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:00:38.283780 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:00:38.287176 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:38.287184 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:38.287199 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:38.287880 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:38.288279 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:00:38.288283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:00:38.288285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:00:38.288504 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:00:38.289030 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:00:38.289036 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:00:38.289384 (+ 348us) client_negotiation.cc:769] Sending connection context
0808 13:00:38.289658 (+ 274us) client_negotiation.cc:240] Negotiation successful
0808 13:00:38.290078 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":419,"threads_started":1}
I20260808 13:00:38.292860 17386 heartbeater.cc:347] Connected to a master server at 127.24.167.62:43679
I20260808 13:00:38.293212 17386 heartbeater.cc:464] Registering TS with master...
I20260808 13:00:38.293972 17386 heartbeater.cc:511] Master 127.24.167.62:43679 requested a full tablet report, sending...
I20260808 13:00:38.295480 16639 ts_manager.cc:194] Registered new tserver with Master: 24db0abfd3c545dd8aec45a422da099e (127.24.167.3:40393)
I20260808 13:00:38.296300 16639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:35903
I20260808 13:00:38.300434 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 13:00:38.318801 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.305153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.24.167.62:43679)
0808 13:00:38.305335 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:38.305344 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:38.305738 (+ 394us) server_negotiation.cc:407] Connection header received
0808 13:00:38.305945 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:38.305951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:38.306026 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:38.306133 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:38.307735 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.308471 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.309934 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.310417 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.310561 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.313134 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:38.313141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:38.313145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:38.313186 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:38.315956 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:38.316687 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:38.316693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:38.316697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:38.316831 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:38.317241 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:38.317246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:38.317249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:38.317591 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:38.317861 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:38.318198 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 13:00:38.318477 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 13:00:38.322983 16639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 13:00:38.329807 16639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 13:00:38.353410 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.344869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.24.167.1:45195)
0808 13:00:38.346288 (+ 1419us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:38.346298 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:38.346328 (+ 30us) server_negotiation.cc:407] Connection header received
0808 13:00:38.346447 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:38.346454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:38.346677 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:38.346873 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:00:38.347445 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.348885 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.351269 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.352205 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.352364 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.352611 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:38.352917 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 13:00:38.353091 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1263,"thread_start_us":1100,"threads_started":1}
I20260808 13:00:38.353461 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.344306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:45195 (local address 127.0.0.1:36932)
0808 13:00:38.345456 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:38.345484 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:38.345627 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:38.346884 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:38.346890 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:38.346916 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:00:38.347266 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:38.347277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.349031 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.349043 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:38.351125 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:38.351136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.352399 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.352405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:38.352610 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.352679 (+ 69us) client_negotiation.cc:769] Sending connection context
0808 13:00:38.352828 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 13:00:38.353013 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"mutex_wait_us":168,"thread_start_us":853,"threads_started":1}
I20260808 13:00:38.353999 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.341859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:36913 (local address 127.0.0.1:45608)
0808 13:00:38.342686 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:38.342753 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:38.342970 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:38.345551 (+ 2581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:38.345561 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:38.345594 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:00:38.346075 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:38.346101 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.348608 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.348613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:38.351163 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:38.351177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.353079 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.353086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:38.353290 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.353321 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 13:00:38.353397 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 13:00:38.353528 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":183,"thread_start_us":401,"threads_started":1}
I20260808 13:00:38.354604 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.341858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:40393 (local address 127.0.0.1:51554)
0808 13:00:38.344615 (+ 2757us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:38.344642 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:38.344853 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:38.345827 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:38.345833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:38.345859 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:00:38.346334 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:38.346343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.348231 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.348238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:38.351694 (+ 3456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:38.351709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.353831 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.353837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:38.354054 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.354089 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 13:00:38.354169 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 13:00:38.354270 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2627,"mutex_wait_us":360,"thread_start_us":2274,"threads_started":1}
I20260808 13:00:38.354871 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.342119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51554 (local address 127.24.167.3:40393)
0808 13:00:38.344115 (+ 1996us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:38.344127 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:38.344743 (+ 616us) server_negotiation.cc:407] Connection header received
0808 13:00:38.345049 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:38.345055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:38.345301 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:38.345483 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:00:38.346520 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.348100 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.351871 (+ 3771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.353330 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.353458 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.353680 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:38.354334 (+ 654us) server_negotiation.cc:299] Negotiation successful
0808 13:00:38.354520 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1778,"thread_start_us":1589,"threads_started":1}
I20260808 13:00:38.356256 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.342147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.24.167.2:36913)
0808 13:00:38.344894 (+ 2747us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:38.344904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:38.344930 (+ 26us) server_negotiation.cc:407] Connection header received
0808 13:00:38.345046 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:38.345055 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:38.345301 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:38.346238 (+ 937us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:00:38.346529 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.348472 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.351361 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.352816 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.355495 (+ 2679us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.355694 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:38.355762 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 13:00:38.355952 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2583,"thread_start_us":2484,"threads_started":1}
I20260808 13:00:38.357903 16860 tablet_service.cc:1467] Processing CreateTablet for tablet 34d54bd889864114a6330804ee57057a (DEFAULT_TABLE table=test-table [id=ffa7f5e02ec542e2ba7c79cdad9dc4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:00:38.358476 17284 tablet_service.cc:1467] Processing CreateTablet for tablet 34d54bd889864114a6330804ee57057a (DEFAULT_TABLE table=test-table [id=ffa7f5e02ec542e2ba7c79cdad9dc4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:00:38.359292 16860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d54bd889864114a6330804ee57057a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:00:38.359956 17284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d54bd889864114a6330804ee57057a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:00:38.359680 17073 tablet_service.cc:1467] Processing CreateTablet for tablet 34d54bd889864114a6330804ee57057a (DEFAULT_TABLE table=test-table [id=ffa7f5e02ec542e2ba7c79cdad9dc4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:00:38.361596 17073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d54bd889864114a6330804ee57057a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:00:38.370991 17412 tablet_bootstrap.cc:492] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33: Bootstrap starting.
I20260808 13:00:38.374855 17412 tablet_bootstrap.cc:654] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33: Neither blocks nor log segments found. Creating new log.
I20260808 13:00:38.375871 17412 log.cc:824] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:38.376907 17414 tablet_bootstrap.cc:492] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e: Bootstrap starting.
I20260808 13:00:38.379464 17415 tablet_bootstrap.cc:492] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa: Bootstrap starting.
I20260808 13:00:38.380952 17414 tablet_bootstrap.cc:654] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e: Neither blocks nor log segments found. Creating new log.
I20260808 13:00:38.380980 17412 tablet_bootstrap.cc:492] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33: No bootstrap required, opened a new log
I20260808 13:00:38.381271 17412 ts_tablet_manager.cc:1397] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260808 13:00:38.382301 17414 log.cc:824] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:38.383268 17415 tablet_bootstrap.cc:654] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa: Neither blocks nor log segments found. Creating new log.
I20260808 13:00:38.384956 17415 log.cc:824] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:38.388325 17415 tablet_bootstrap.cc:492] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa: No bootstrap required, opened a new log
I20260808 13:00:38.388652 17415 ts_tablet_manager.cc:1397] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260808 13:00:38.388991 17414 tablet_bootstrap.cc:492] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e: No bootstrap required, opened a new log
I20260808 13:00:38.389267 17414 ts_tablet_manager.cc:1397] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260808 13:00:38.390821 17412 raft_consensus.cc:348] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } } peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } }
I20260808 13:00:38.391320 17412 raft_consensus.cc:374] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:00:38.391489 17412 raft_consensus.cc:729] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c22d98dda44b5ebd1ac35bd84dbd33, State: Initialized, Role: FOLLOWER
I20260808 13:00:38.392191 17412 consensus_queue.cc:260] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } } peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } }
I20260808 13:00:38.394295 17412 ts_tablet_manager.cc:1428] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260808 13:00:38.398108 17415 raft_consensus.cc:348] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } } peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } }
I20260808 13:00:38.398711 17415 raft_consensus.cc:374] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:00:38.398845 17415 raft_consensus.cc:729] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbed5731bdde46f281e4c55af33c61fa, State: Initialized, Role: FOLLOWER
I20260808 13:00:38.399528 17415 consensus_queue.cc:260] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } } peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } }
I20260808 13:00:38.401073 17170 heartbeater.cc:503] Master 127.24.167.62:43679 was elected leader, sending a full tablet report...
I20260808 13:00:38.400795 17414 raft_consensus.cc:348] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } } peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } }
I20260808 13:00:38.401290 17414 raft_consensus.cc:374] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:00:38.401404 17414 raft_consensus.cc:729] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24db0abfd3c545dd8aec45a422da099e, State: Initialized, Role: FOLLOWER
I20260808 13:00:38.401612 17415 ts_tablet_manager.cc:1428] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260808 13:00:38.402083 17414 consensus_queue.cc:260] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } } peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } }
I20260808 13:00:38.403342 17386 heartbeater.cc:503] Master 127.24.167.62:43679 was elected leader, sending a full tablet report...
I20260808 13:00:38.403827 17414 ts_tablet_manager.cc:1428] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260808 13:00:38.404991 17421 raft_consensus.cc:482] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 13:00:38.405278 17421 raft_consensus.cc:504] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } } peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } }
I20260808 13:00:38.407358 17421 leader_election.cc:288] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24db0abfd3c545dd8aec45a422da099e (127.24.167.3:40393), fbed5731bdde46f281e4c55af33c61fa (127.24.167.2:36913)
I20260808 13:00:38.415403 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.407863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:45659 (local address 127.24.167.3:40393)
0808 13:00:38.408333 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:38.408337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:38.409929 (+ 1592us) server_negotiation.cc:407] Connection header received
0808 13:00:38.410173 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:38.410179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:38.410248 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:38.410364 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:00:38.411046 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.411825 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.413614 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.414449 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.414644 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.414720 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:38.415020 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 13:00:38.415150 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329}
I20260808 13:00:38.415609 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.407703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:40393 (local address 127.24.167.1:45659)
0808 13:00:38.409820 (+ 2117us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:38.409838 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:38.410031 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:38.410429 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:38.410437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:38.410467 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:00:38.410894 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:38.410904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.411966 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.411972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:38.413430 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:38.413443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.414656 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.414662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:38.414843 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.414869 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 13:00:38.414991 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 13:00:38.415126 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1962,"spinlock_wait_cycles":19200,"thread_start_us":1752,"threads_started":1}
I20260808 13:00:38.416324 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.408553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:55635 (local address 127.24.167.2:36913)
0808 13:00:38.408745 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:38.408751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:38.410517 (+ 1766us) server_negotiation.cc:407] Connection header received
0808 13:00:38.410810 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:38.410816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:38.410899 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:38.411006 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:00:38.411572 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.412360 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.414238 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.415196 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.415385 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.415483 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:38.415894 (+ 411us) server_negotiation.cc:299] Negotiation successful
0808 13:00:38.416065 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 13:00:38.416337 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.408017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:36913 (local address 127.24.167.1:55635)
0808 13:00:38.410358 (+ 2341us) negotiation.cc:107] Waiting for socket to connect
0808 13:00:38.410429 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 13:00:38.410654 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:00:38.411021 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:00:38.411026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:00:38.411054 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:00:38.411408 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:38.411419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.412529 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.412535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:38.414066 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:00:38.414080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.415454 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.415463 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:00:38.415658 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.415699 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 13:00:38.415850 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 13:00:38.416025 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2210,"thread_start_us":1958,"threads_started":1}
I20260808 13:00:38.416816 17316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d54bd889864114a6330804ee57057a" candidate_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24db0abfd3c545dd8aec45a422da099e" is_pre_election: true
I20260808 13:00:38.417394 17316 raft_consensus.cc:2454] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36c22d98dda44b5ebd1ac35bd84dbd33 in term 0.
I20260808 13:00:38.418156 16776 leader_election.cc:302] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24db0abfd3c545dd8aec45a422da099e, 36c22d98dda44b5ebd1ac35bd84dbd33; no voters:
I20260808 13:00:38.418740 17421 raft_consensus.cc:2781] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 13:00:38.418923 17421 raft_consensus.cc:482] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 13:00:38.419050 17421 raft_consensus.cc:3037] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:00:38.418788 17098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d54bd889864114a6330804ee57057a" candidate_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbed5731bdde46f281e4c55af33c61fa" is_pre_election: true
I20260808 13:00:38.419420 17098 raft_consensus.cc:2454] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36c22d98dda44b5ebd1ac35bd84dbd33 in term 0.
I20260808 13:00:38.421659 17421 raft_consensus.cc:504] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } } peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } }
I20260808 13:00:38.422400 17421 leader_election.cc:288] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [CANDIDATE]: Term 1 election: Requested vote from peers 24db0abfd3c545dd8aec45a422da099e (127.24.167.3:40393), fbed5731bdde46f281e4c55af33c61fa (127.24.167.2:36913)
I20260808 13:00:38.422899 17316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d54bd889864114a6330804ee57057a" candidate_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24db0abfd3c545dd8aec45a422da099e"
I20260808 13:00:38.423063 17098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d54bd889864114a6330804ee57057a" candidate_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbed5731bdde46f281e4c55af33c61fa"
I20260808 13:00:38.423170 17316 raft_consensus.cc:3037] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:00:38.423306 17098 raft_consensus.cc:3037] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:00:38.425545 17316 raft_consensus.cc:2454] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36c22d98dda44b5ebd1ac35bd84dbd33 in term 1.
I20260808 13:00:38.425657 17098 raft_consensus.cc:2454] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36c22d98dda44b5ebd1ac35bd84dbd33 in term 1.
I20260808 13:00:38.426017 16776 leader_election.cc:302] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24db0abfd3c545dd8aec45a422da099e, 36c22d98dda44b5ebd1ac35bd84dbd33; no voters:
I20260808 13:00:38.426469 17421 raft_consensus.cc:2781] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:00:38.427030 17421 raft_consensus.cc:686] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 1 LEADER]: Becoming Leader. State: Replica: 36c22d98dda44b5ebd1ac35bd84dbd33, State: Running, Role: LEADER
I20260808 13:00:38.427561 17421 consensus_queue.cc:237] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } } peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } }
W20260808 13:00:38.428931 17175 tablet.cc:2369] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:00:38.432873 16639 catalog_manager.cc:5495] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36c22d98dda44b5ebd1ac35bd84dbd33 (127.24.167.1). New cstate: current_term: 1 leader_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } health_report { overall_health: HEALTHY } } }
I20260808 13:00:38.459879 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.453682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.24.167.1:45195)
0808 13:00:38.453904 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:38.453909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:38.453927 (+ 18us) server_negotiation.cc:407] Connection header received
0808 13:00:38.454021 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:38.454024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:38.454124 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:38.454300 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:00:38.454724 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.455477 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.457595 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.458017 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.458116 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.458728 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:00:38.459279 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:00:38.459394 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:38.459534 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 13:00:38.459605 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 13:00:38.468746 17098 raft_consensus.cc:1261] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa [term 1 FOLLOWER]: Refusing update from remote peer 36c22d98dda44b5ebd1ac35bd84dbd33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:00:38.468870 17316 raft_consensus.cc:1261] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e [term 1 FOLLOWER]: Refusing update from remote peer 36c22d98dda44b5ebd1ac35bd84dbd33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:00:38.469789 17431 consensus_queue.cc:1035] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:00:38.470189 17421 consensus_queue.cc:1035] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:00:38.482190 17438 mvcc.cc:205] Tried to move back new op lower bound from 7316250781557456896 to 7316250781402992640. Current Snapshot: MvccSnapshot[applied={T|T < 7316250781557456896}]
I20260808 13:00:38.486871 17440 mvcc.cc:205] Tried to move back new op lower bound from 7316250781557456896 to 7316250781402992640. Current Snapshot: MvccSnapshot[applied={T|T < 7316250781557456896}]
I20260808 13:00:38.499042 16639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:00:38.499367 16639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:00:38.503813 16639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Sending DeleteTablet for 3 replicas of tablet 34d54bd889864114a6330804ee57057a
I20260808 13:00:38.505100 17284 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d54bd889864114a6330804ee57057a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:00:38 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:51554
I20260808 13:00:38.505383 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 16745
I20260808 13:00:38.505371 17073 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d54bd889864114a6330804ee57057a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:00:38 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:45608
I20260808 13:00:38.506887 16860 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d54bd889864114a6330804ee57057a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:00:38 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:36932
I20260808 13:00:38.518231 17452 tablet_replica.cc:330] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e: stopping tablet replica
I20260808 13:00:38.518901 17452 raft_consensus.cc:2229] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:00:38.519591 17452 raft_consensus.cc:2258] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:00:38.524995 17452 ts_tablet_manager.cc:1905] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:00:38.525820 17455 tablet_replica.cc:330] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33: stopping tablet replica
I20260808 13:00:38.526415 17455 raft_consensus.cc:2229] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 1 LEADER]: Raft consensus shutting down.
I20260808 13:00:38.528023 17453 tablet_replica.cc:330] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa: stopping tablet replica
I20260808 13:00:38.528910 17455 raft_consensus.cc:2258] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:00:38.529496 17453 raft_consensus.cc:2229] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:00:38.530401 17453 raft_consensus.cc:2258] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:00:38.532533 17455 ts_tablet_manager.cc:1905] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:00:38.533509 17452 ts_tablet_manager.cc:1918] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:00:38.533751 17452 log.cc:1196] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/34d54bd889864114a6330804ee57057a
I20260808 13:00:38.534433 17452 ts_tablet_manager.cc:1939] T 34d54bd889864114a6330804ee57057a P 24db0abfd3c545dd8aec45a422da099e: Deleting consensus metadata
I20260808 13:00:38.534583 17453 ts_tablet_manager.cc:1905] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:00:38.536562 16618 catalog_manager.cc:4841] TS 24db0abfd3c545dd8aec45a422da099e (127.24.167.3:40393): tablet 34d54bd889864114a6330804ee57057a (table test-table [id=ffa7f5e02ec542e2ba7c79cdad9dc4da]) successfully deleted
Aug 08 13:00:38 dist-test-slave-tk9z krb5kdc[16570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 13:00:38.542299 17455 ts_tablet_manager.cc:1918] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:00:38.542567 17455 log.cc:1196] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/34d54bd889864114a6330804ee57057a
I20260808 13:00:38.543256 17455 ts_tablet_manager.cc:1939] T 34d54bd889864114a6330804ee57057a P 36c22d98dda44b5ebd1ac35bd84dbd33: Deleting consensus metadata
I20260808 13:00:38.543838 17453 ts_tablet_manager.cc:1918] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:00:38.544073 17453 log.cc:1196] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/34d54bd889864114a6330804ee57057a
I20260808 13:00:38.544689 17453 ts_tablet_manager.cc:1939] T 34d54bd889864114a6330804ee57057a P fbed5731bdde46f281e4c55af33c61fa: Deleting consensus metadata
I20260808 13:00:38.545378 16620 catalog_manager.cc:4841] TS 36c22d98dda44b5ebd1ac35bd84dbd33 (127.24.167.1:45195): tablet 34d54bd889864114a6330804ee57057a (table test-table [id=ffa7f5e02ec542e2ba7c79cdad9dc4da]) successfully deleted
I20260808 13:00:38.546684 16620 catalog_manager.cc:4841] TS fbed5731bdde46f281e4c55af33c61fa (127.24.167.2:36913): tablet 34d54bd889864114a6330804ee57057a (table test-table [id=ffa7f5e02ec542e2ba7c79cdad9dc4da]) successfully deleted
I20260808 13:00:38.551123 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.507626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36944 (local address 127.24.167.1:45195)
0808 13:00:38.507857 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:38.507867 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:38.527282 (+ 19415us) server_negotiation.cc:407] Connection header received
0808 13:00:38.527508 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:38.527514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:38.527629 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:38.527755 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:38.530159 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.531470 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.533645 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.534416 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.534545 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.543971 (+ 9426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:38.543990 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:38.544009 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:38.544080 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:38.547876 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:38.548894 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:38.548901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:38.548903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:38.549080 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:38.549524 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:38.549527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:38.549529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:38.549885 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:38.550110 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:38.550593 (+ 483us) server_negotiation.cc:299] Negotiation successful
0808 13:00:38.550782 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 13:00:38.551842 16954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:38.653190 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 16745
I20260808 13:00:38.680734 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 16976
Aug 08 13:00:38 dist-test-slave-tk9z krb5kdc[16570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 13:00:38.698798 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.681593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.24.167.2:36913)
0808 13:00:38.681770 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:38.681774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:38.681897 (+ 123us) server_negotiation.cc:407] Connection header received
0808 13:00:38.682074 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:38.682078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:38.682143 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:38.682238 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:38.683957 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.684739 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.686828 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.687244 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.687371 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.692657 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:38.692673 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:38.692687 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:38.692728 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:38.695864 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:38.696599 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:38.696606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:38.696608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:38.696729 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:38.697139 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:38.697144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:38.697146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:38.697552 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:38.697874 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:38.698184 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 13:00:38.698438 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 13:00:38.699522 17165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:38.790992 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 16976
I20260808 13:00:38.816272 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 17183
Aug 08 13:00:38 dist-test-slave-tk9z krb5kdc[16570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 13:00:38.832310 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:00:38.817094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.24.167.3:40393)
0808 13:00:38.817264 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 13:00:38.817269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:00:38.817288 (+ 19us) server_negotiation.cc:407] Connection header received
0808 13:00:38.817438 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:00:38.817463 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:00:38.817544 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:00:38.817632 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:00:38.819149 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.819963 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.821471 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:00:38.821863 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:00:38.821963 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:00:38.826921 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:00:38.826942 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:00:38.826966 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:00:38.827011 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:00:38.829670 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:38.830414 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:38.830427 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:38.830434 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:38.830621 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:00:38.830970 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:00:38.830973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:00:38.830975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:00:38.831341 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:00:38.831582 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 13:00:38.831859 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 13:00:38.832036 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 13:00:38.833043 17376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:38.927356 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 17183
I20260808 13:00:38.951642 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 16591
I20260808 13:00:38.952939 16684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:00:39.066704 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 16591
I20260808 13:00:59.097648 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:43679
--webserver_interface=127.24.167.62
--webserver_port=38055
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:43679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:00:59.470117 17463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:00:59.470523 17463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:00:59.470698 17463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:00:59.478024 17463 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 13:00:59.478135 17463 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 13:00:59.478215 17463 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 13:00:59.478302 17463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:00:59.478415 17463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:00:59.478530 17463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:00:59.478590 17463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:00:59.478669 17463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:00:59.489446 17463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:43679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:43679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=38055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:00:59.490926 17463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:00:59.492578 17463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:00:59.519119 17477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:59.519237 17479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:00:59.519124 17474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:00:59.520066 17463 server_base.cc:1034] running on GCE node
I20260808 13:00:59.521023 17463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:00:59.522764 17463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:00:59.524031 17463 hybrid_clock.cc:648] HybridClock initialized: now 1786194059523964 us; error 61 us; skew 500 ppm
I20260808 13:00:59.524448 17463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:00:59 dist-test-slave-tk9z krb5kdc[16570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194059, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:00:59.533207 17463 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:00:59.535238 17463 webserver.cc:460] Webserver started at http://127.24.167.62:38055/ using document root <none> and password file <none>
I20260808 13:00:59.536124 17463 fs_manager.cc:359] Metadata directory not provided
I20260808 13:00:59.536291 17463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:00:59.542732 17463 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 13:00:59.546124 17488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:00:59.547717 17463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:00:59.547950 17463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "d7335399a36e424da32d6a3b5c8c32a4"
format_stamp: "Formatted at 2026-08-08 13:00:36 on dist-test-slave-tk9z"
server_key: "d7f1c3616429fc452efd402e0ece60a7"
server_key_iv: "76ecc67eb9f5b27fac1edc61f9466a31"
server_key_version: "encryptionkey@0"
I20260808 13:00:59.549152 17463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:00:59.576973 17463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:00:59.582245 17463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:00:59.582651 17463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:00:59.603610 17463 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:43679
I20260808 13:00:59.603657 17560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:43679 every 8 connection(s)
I20260808 13:00:59.605237 17463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:00:59.612682 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 17463
I20260808 13:00:59.612870 17561 sys_catalog.cc:263] Verifying existing consensus state
I20260808 13:00:59.613611 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:45195
--local_ip_for_outbound_sockets=127.24.167.1
--tserver_master_addrs=127.24.167.62:43679
--webserver_port=45813
--webserver_interface=127.24.167.1
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 13:00:59.616482 17561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Bootstrap starting.
I20260808 13:00:59.638247 17561 log.cc:824] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Log is configured to *not* fsync() on all Append() calls
I20260808 13:00:59.652369 17561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 13:00:59.653071 17561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Bootstrap complete.
I20260808 13:00:59.664269 17561 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } }
I20260808 13:00:59.665264 17561 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7335399a36e424da32d6a3b5c8c32a4, State: Initialized, Role: FOLLOWER
I20260808 13:00:59.666014 17561 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } }
I20260808 13:00:59.666308 17561 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:00:59.666476 17561 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:00:59.666705 17561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 13:00:59.671177 17561 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } }
I20260808 13:00:59.671809 17561 leader_election.cc:302] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7335399a36e424da32d6a3b5c8c32a4; no voters:
I20260808 13:00:59.672634 17561 leader_election.cc:288] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 13:00:59.672814 17568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 13:00:59.674140 17568 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 2 LEADER]: Becoming Leader. State: Replica: d7335399a36e424da32d6a3b5c8c32a4, State: Running, Role: LEADER
I20260808 13:00:59.674917 17568 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } }
I20260808 13:00:59.675457 17561 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:00:59.677597 17572 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7335399a36e424da32d6a3b5c8c32a4. Latest consensus state: current_term: 2 leader_uuid: "d7335399a36e424da32d6a3b5c8c32a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } } }
I20260808 13:00:59.678017 17572 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [sys.catalog]: This master's current role is: LEADER
I20260808 13:00:59.678647 17571 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d7335399a36e424da32d6a3b5c8c32a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } } }
I20260808 13:00:59.678970 17571 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [sys.catalog]: This master's current role is: LEADER
I20260808 13:00:59.686987 17580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:00:59.697218 17580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:00:59.699152 17580 catalog_manager.cc:1259] Loaded cluster ID: 80bbd6c7076446359f9faf247c4ce482
I20260808 13:00:59.699277 17580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:00:59.705053 17580 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:00:59.707036 17580 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Loaded TSK: 0
I20260808 13:00:59.739034 17580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Generated new TSK 1
I20260808 13:00:59.739480 17580 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 13:01:00.008837 17564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:00.009182 17564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:00.009354 17564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:00.017191 17564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:00.017383 17564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:00.017565 17564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 13:01:00.027976 17564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:45195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=45813
--tserver_master_addrs=127.24.167.62:43679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:00.029435 17564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:00.030930 17564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:00.047672 17612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:00.047876 17615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:00.048200 17610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:00.048626 17564 server_base.cc:1034] running on GCE node
I20260808 13:01:00.049435 17564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:00.050968 17564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:00.052279 17564 hybrid_clock.cc:648] HybridClock initialized: now 1786194060052149 us; error 88 us; skew 500 ppm
I20260808 13:01:00.052740 17564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:00 dist-test-slave-tk9z krb5kdc[16570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194060, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:00.057554 17564 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 13:01:00.059530 17564 webserver.cc:460] Webserver started at http://127.24.167.1:45813/ using document root <none> and password file <none>
I20260808 13:01:00.060456 17564 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:00.060618 17564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:00.068192 17564 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 13:01:00.071749 17628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:00.073532 17564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 13:01:00.073715 17564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "36c22d98dda44b5ebd1ac35bd84dbd33"
format_stamp: "Formatted at 2026-08-08 13:00:37 on dist-test-slave-tk9z"
server_key: "43c0cd2965b824e8d3055229ea7f594c"
server_key_iv: "0bc28cd3f579af634199aa24a844ffa6"
server_key_version: "encryptionkey@0"
I20260808 13:01:00.075280 17564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:00.110725 17564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:00.116786 17564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:00.117120 17564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:00.118438 17564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:00.120338 17564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:00.120466 17564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:00.120580 17564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:00.120666 17564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:00.159520 17564 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:45195
I20260808 13:01:00.159610 17797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:45195 every 8 connection(s)
I20260808 13:01:00.161226 17564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 13:01:00.165628 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 17564
I20260808 13:01:00.166428 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:36913
--local_ip_for_outbound_sockets=127.24.167.2
--tserver_master_addrs=127.24.167.62:43679
--webserver_port=34853
--webserver_interface=127.24.167.2
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 13:01:00 dist-test-slave-tk9z krb5kdc[16570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194060, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:01:00.190897 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:00.164623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:50211)
0808 13:01:00.165556 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:00.165603 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:00.167625 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:00.169452 (+ 1827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:00.169460 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:00.169999 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:00.170638 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:00.170657 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:00.173343 (+ 2686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:00.173347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:00.175074 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:00.175087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:00.176753 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:00.176759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:00.177020 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:00.177782 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:01:00.177828 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:00.180866 (+ 3038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:00.186043 (+ 5177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:00.186054 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:00.186071 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:00.186710 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:00.187293 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:00.187297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:00.187299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:00.187492 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:00.188584 (+ 1092us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:00.188591 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:00.189035 (+ 444us) client_negotiation.cc:769] Sending connection context
0808 13:01:00.189333 (+ 298us) client_negotiation.cc:240] Negotiation successful
0808 13:01:00.189759 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":392,"threads_started":1}
I20260808 13:01:00.191349 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:00.164321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:50211 (local address 127.24.167.62:43679)
0808 13:01:00.164950 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:00.164989 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:00.165768 (+ 779us) server_negotiation.cc:407] Connection header received
0808 13:01:00.168279 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:00.168303 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:00.168844 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:00.169291 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:00.170901 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:00.173126 (+ 2225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:00.175235 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:00.176477 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:00.176700 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:00.181295 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:00.181308 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:00.181331 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:00.181387 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:00.185843 (+ 4456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:00.186887 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:00.186899 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:00.186910 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:00.187140 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:00.187637 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:00.187645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:00.187650 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:00.188412 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:00.188642 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:00.189620 (+ 978us) server_negotiation.cc:299] Negotiation successful
0808 13:01:00.190074 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":273,"threads_started":1}
I20260808 13:01:00.194298 17799 heartbeater.cc:347] Connected to a master server at 127.24.167.62:43679
I20260808 13:01:00.194716 17799 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:00.195737 17799 heartbeater.cc:511] Master 127.24.167.62:43679 requested a full tablet report, sending...
I20260808 13:01:00.198278 17511 ts_manager.cc:194] Registered new tserver with Master: 36c22d98dda44b5ebd1ac35bd84dbd33 (127.24.167.1:45195)
I20260808 13:01:00.199798 17511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:50211
W20260808 13:01:00.525035 17803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:00.525306 17803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:00.525481 17803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:00.532725 17803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:00.532936 17803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:00.533217 17803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 13:01:00.544755 17803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:36913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=34853
--tserver_master_addrs=127.24.167.62:43679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:00.546222 17803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:00.547699 17803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:00.561903 17812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:00.561908 17814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:00.562590 17803 server_base.cc:1034] running on GCE node
W20260808 13:01:00.562682 17810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:00.563386 17803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:00.564289 17803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:00.565538 17803 hybrid_clock.cc:648] HybridClock initialized: now 1786194060565495 us; error 54 us; skew 500 ppm
I20260808 13:01:00.566038 17803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:00 dist-test-slave-tk9z krb5kdc[16570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194060, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:00.569305 17803 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 13:01:00.571000 17803 webserver.cc:460] Webserver started at http://127.24.167.2:34853/ using document root <none> and password file <none>
I20260808 13:01:00.571733 17803 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:00.571882 17803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:00.577517 17803 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 13:01:00.580379 17824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:00.581861 17803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 13:01:00.582036 17803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "fbed5731bdde46f281e4c55af33c61fa"
format_stamp: "Formatted at 2026-08-08 13:00:37 on dist-test-slave-tk9z"
server_key: "04ed2b600a808f6d2dcdfd2b8d3f736f"
server_key_iv: "6cebe99f537bcea6bc68a0daf5f6932e"
server_key_version: "encryptionkey@0"
I20260808 13:01:00.583154 17803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:00.615437 17803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:00.619882 17803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:00.620180 17803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:00.621289 17803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:00.622699 17803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:00.622814 17803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:00.622956 17803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:00.623046 17803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:00.657002 17803 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:36913
I20260808 13:01:00.657095 18008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:36913 every 8 connection(s)
I20260808 13:01:00.658596 17803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 13:01:00.662007 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 17803
I20260808 13:01:00.662926 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:40393
--local_ip_for_outbound_sockets=127.24.167.3
--tserver_master_addrs=127.24.167.62:43679
--webserver_port=44311
--webserver_interface=127.24.167.3
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 13:01:00 dist-test-slave-tk9z krb5kdc[16570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194060, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:01:00.681598 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:00.661343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:56539 (local address 127.24.167.62:43679)
0808 13:01:00.661547 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:00.661551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:00.663272 (+ 1721us) server_negotiation.cc:407] Connection header received
0808 13:01:00.665061 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:00.665067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:00.665198 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:00.665320 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:00.667022 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:00.667762 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:00.669357 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:00.669815 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:00.669955 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:00.674076 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:00.674084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:00.674090 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:00.674143 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:00.677956 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:00.678861 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:00.678869 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:00.678875 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:00.679053 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:00.679491 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:00.679499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:00.679505 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:00.679949 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:00.680107 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:00.680828 (+ 721us) server_negotiation.cc:299] Negotiation successful
0808 13:01:00.681135 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 13:01:00.682709 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:00.662043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:56539)
0808 13:01:00.662885 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:00.662922 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:00.664760 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:00.665578 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:00.665587 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:00.666132 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:00.666836 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:00.666856 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:00.667923 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:00.667937 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:00.669205 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:00.669218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:00.670073 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:00.670080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:00.670333 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:00.671119 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:01:00.671143 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:00.673765 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:00.678113 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:00.678121 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:00.678136 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:00.678703 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:00.679170 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:00.679173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:00.679175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:00.679370 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:00.680072 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:00.680077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:00.680419 (+ 342us) client_negotiation.cc:769] Sending connection context
0808 13:01:00.680650 (+ 231us) client_negotiation.cc:240] Negotiation successful
0808 13:01:00.681269 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":378,"threads_started":1}
I20260808 13:01:00.684350 18009 heartbeater.cc:347] Connected to a master server at 127.24.167.62:43679
I20260808 13:01:00.684720 18009 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:00.685688 18009 heartbeater.cc:511] Master 127.24.167.62:43679 requested a full tablet report, sending...
I20260808 13:01:00.687580 17511 ts_manager.cc:194] Registered new tserver with Master: fbed5731bdde46f281e4c55af33c61fa (127.24.167.2:36913)
I20260808 13:01:00.688870 17511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:56539
W20260808 13:01:01.052861 18013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:01.053223 18013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:01.053440 18013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:01.061012 18013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:01.061128 18013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:01.061348 18013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 13:01:01.071703 18013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:40393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=44311
--tserver_master_addrs=127.24.167.62:43679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:01.073100 18013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:01.074568 18013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:01.089365 18020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:01.089267 18018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:01.089262 18024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:01.090601 18013 server_base.cc:1034] running on GCE node
I20260808 13:01:01.091323 18013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:01.092398 18013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:01.093633 18013 hybrid_clock.cc:648] HybridClock initialized: now 1786194061093581 us; error 38 us; skew 500 ppm
I20260808 13:01:01.094009 18013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:01 dist-test-slave-tk9z krb5kdc[16570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:01.097340 18013 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 13:01:01.098949 18013 webserver.cc:460] Webserver started at http://127.24.167.3:44311/ using document root <none> and password file <none>
I20260808 13:01:01.099680 18013 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:01.099841 18013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:01.105700 18013 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 13:01:01.108659 18034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:01.110041 18013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 13:01:01.110231 18013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "24db0abfd3c545dd8aec45a422da099e"
format_stamp: "Formatted at 2026-08-08 13:00:38 on dist-test-slave-tk9z"
server_key: "8b3288ea8ec68d1d318ed0e6fc6f1b47"
server_key_iv: "cf026565e609df9f308435af70268ab0"
server_key_version: "encryptionkey@0"
I20260808 13:01:01.111317 18013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:01.126797 18013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:01.131121 18013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:01.131430 18013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:01.132488 18013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:01.133879 18013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:01.133980 18013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:01.134136 18013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:01.134214 18013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:01.169981 18013 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:40393
I20260808 13:01:01.170047 18217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:40393 every 8 connection(s)
I20260808 13:01:01.171545 18013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 13:01:01.179329 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 18013
Aug 08 13:01:01 dist-test-slave-tk9z krb5kdc[16570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:01:01.195779 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.174219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:40115 (local address 127.24.167.62:43679)
0808 13:01:01.174477 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:01.174484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:01.175741 (+ 1257us) server_negotiation.cc:407] Connection header received
0808 13:01:01.177765 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:01.177769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:01.177841 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:01.177941 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:01.179947 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.181017 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.183247 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.183682 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.183833 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.188572 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:01.188581 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:01.188587 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:01.188632 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:01.192226 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:01.193040 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:01.193046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:01.193050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:01.193184 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:01.193671 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:01.193678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:01.193686 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:01.194002 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:01.194172 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:01.195132 (+ 960us) server_negotiation.cc:299] Negotiation successful
0808 13:01:01.195310 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 13:01:01.196986 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.174730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:40115)
0808 13:01:01.175564 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:01.175606 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:01.177497 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:01.178181 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:01.178190 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:01.178838 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:01.179732 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:01.179762 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.181192 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.181197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:01.183099 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:01.183111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.183927 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.183936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:01.184205 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.184986 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:01:01.185018 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:01.188298 (+ 3280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:01.192393 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:01.192403 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:01.192434 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:01.192896 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:01.193305 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:01.193310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:01.193314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:01.193539 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:01.194141 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:01.194150 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:01.194633 (+ 483us) client_negotiation.cc:769] Sending connection context
0808 13:01:01.194991 (+ 358us) client_negotiation.cc:240] Negotiation successful
0808 13:01:01.195589 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":291,"threads_started":1}
I20260808 13:01:01.197758 18225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.180435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42766 (local address 127.24.167.62:43679)
0808 13:01:01.180950 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:01.180956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:01.181315 (+ 359us) server_negotiation.cc:407] Connection header received
0808 13:01:01.181451 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:01.181456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:01.181535 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:01.181650 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:01.183745 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.184609 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.186845 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.187691 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.187826 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.191588 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:01.191593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:01.191597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:01.191647 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:01.194542 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:01.195471 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:01.195477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:01.195481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:01.195626 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:01.196188 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:01.196193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:01.196195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:01.196466 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:01.196667 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:01.197107 (+ 440us) server_negotiation.cc:299] Negotiation successful
0808 13:01:01.197344 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":251,"threads_started":1}
I20260808 13:01:01.198750 18218 heartbeater.cc:347] Connected to a master server at 127.24.167.62:43679
I20260808 13:01:01.199141 18218 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:01.200064 18218 heartbeater.cc:511] Master 127.24.167.62:43679 requested a full tablet report, sending...
I20260808 13:01:01.201498 17511 ts_manager.cc:194] Registered new tserver with Master: 24db0abfd3c545dd8aec45a422da099e (127.24.167.3:40393)
I20260808 13:01:01.202359 17511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:40115
I20260808 13:01:01.203540 17799 heartbeater.cc:503] Master 127.24.167.62:43679 was elected leader, sending a full tablet report...
I20260808 13:01:01.213359 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 13:01:01.220964 18225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.214341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42778 (local address 127.24.167.62:43679)
0808 13:01:01.214586 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:01.214590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:01.215158 (+ 568us) server_negotiation.cc:407] Connection header received
0808 13:01:01.215482 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:01.215486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:01.215709 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:01.215877 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:01:01.216414 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.217136 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.218749 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.219198 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.219347 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.219916 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:01:01.220223 (+ 307us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 13:01:01.220667 (+ 444us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42778: token expired
Metrics: {"server-negotiator.queue_time_us":118}
W20260808 13:01:01.221163 18225 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42778: token expired
W20260808 13:01:01.221822 17397 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.24.167.62:43679)
I20260808 13:01:01.235837 18225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.222522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.24.167.62:43679)
0808 13:01:01.222720 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:01.222724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:01.222748 (+ 24us) server_negotiation.cc:407] Connection header received
0808 13:01:01.222790 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:01.222794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:01.222853 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:01.222966 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:01.224492 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.225219 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.226633 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.227056 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.227153 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.230001 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:01.230005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:01.230008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:01.230044 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:01.232865 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:01.233658 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:01.233662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:01.233664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:01.233826 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:01.234327 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:01.234331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:01.234333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:01.234786 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:01.235019 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:01.235316 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 13:01:01.235521 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 13:01:01.239871 17511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 13:01:01.247777 17511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 13:01:01.274746 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.261599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.24.167.3:40393)
0808 13:01:01.264136 (+ 2537us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:01.264149 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:01.264215 (+ 66us) server_negotiation.cc:407] Connection header received
0808 13:01:01.264338 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:01.264346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:01.264624 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:01.264825 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:01.266014 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.267828 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.271376 (+ 3548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.273094 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.273264 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.273553 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:01.274138 (+ 585us) server_negotiation.cc:299] Negotiation successful
0808 13:01:01.274326 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2340,"thread_start_us":3101,"threads_started":1}
I20260808 13:01:01.274739 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.261093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:40393 (local address 127.0.0.1:44206)
0808 13:01:01.263954 (+ 2861us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:01.263985 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:01.264127 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:01.265193 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:01.265204 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:01.265256 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:01.265762 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:01.265779 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.268033 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.268038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:01.271178 (+ 3140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:01.271192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.273717 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.273724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:01.273920 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.273952 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 13:01:01.274046 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 13:01:01.274163 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2696,"mutex_wait_us":46,"thread_start_us":1129,"threads_started":1}
I20260808 13:01:01.274761 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.260912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:45195 (local address 127.0.0.1:43344)
0808 13:01:01.262871 (+ 1959us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:01.262935 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:01.263187 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:01.266980 (+ 3793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:01.266985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:01.267016 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:01.267395 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:01.267407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.269277 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.269283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:01.271395 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:01.271410 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.273423 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.273431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:01.273660 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.273738 (+ 78us) client_negotiation.cc:769] Sending connection context
0808 13:01:01.273877 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 13:01:01.274103 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1600,"mutex_wait_us":102,"spinlock_wait_cycles":3328,"thread_start_us":2664,"threads_started":1}
I20260808 13:01:01.275895 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.264037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:36913 (local address 127.0.0.1:52698)
0808 13:01:01.267018 (+ 2981us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:01.267036 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:01.267173 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:01.267815 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:01.267819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:01.267845 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:01.268381 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:01.268394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.270663 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.270670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:01.272329 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:01.272341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.275006 (+ 2665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.275014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:01.275185 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.275221 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 13:01:01.275319 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 13:01:01.275479 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2853,"mutex_wait_us":23,"thread_start_us":2673,"threads_started":1}
I20260808 13:01:01.275983 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.264565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52698 (local address 127.24.167.2:36913)
0808 13:01:01.266547 (+ 1982us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:01.266555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:01.267260 (+ 705us) server_negotiation.cc:407] Connection header received
0808 13:01:01.267368 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:01.267373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:01.267589 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:01.267804 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:01.268579 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.270509 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.272718 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.274475 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.274624 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.274816 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:01.275414 (+ 598us) server_negotiation.cc:299] Negotiation successful
0808 13:01:01.275615 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1837,"thread_start_us":1670,"threads_started":1}
I20260808 13:01:01.276700 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.261604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.24.167.1:45195)
0808 13:01:01.266189 (+ 4585us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:01.266200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:01.266235 (+ 35us) server_negotiation.cc:407] Connection header received
0808 13:01:01.266398 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:01.266408 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:01.266687 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:01.266984 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:01.267597 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.269104 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.271584 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.273116 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.275813 (+ 2697us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.276067 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:01.276160 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 13:01:01.276362 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4393,"thread_start_us":4255,"threads_started":1}
I20260808 13:01:01.279775 17903 tablet_service.cc:1467] Processing CreateTablet for tablet e98aaf4cd2524dbc96a3435ef4619132 (DEFAULT_TABLE table=test-table [id=5f97fb61e5fd46c986e5c203f9741eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:01.279781 18114 tablet_service.cc:1467] Processing CreateTablet for tablet e98aaf4cd2524dbc96a3435ef4619132 (DEFAULT_TABLE table=test-table [id=5f97fb61e5fd46c986e5c203f9741eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:01.280334 17695 tablet_service.cc:1467] Processing CreateTablet for tablet e98aaf4cd2524dbc96a3435ef4619132 (DEFAULT_TABLE table=test-table [id=5f97fb61e5fd46c986e5c203f9741eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:01.281271 17903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e98aaf4cd2524dbc96a3435ef4619132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:01.281345 18114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e98aaf4cd2524dbc96a3435ef4619132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:01.281728 17695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e98aaf4cd2524dbc96a3435ef4619132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:01.296267 18249 tablet_bootstrap.cc:492] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa: Bootstrap starting.
I20260808 13:01:01.299943 18249 tablet_bootstrap.cc:654] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:01.301783 18249 log.cc:824] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:01.303508 18252 tablet_bootstrap.cc:492] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e: Bootstrap starting.
I20260808 13:01:01.309157 18252 tablet_bootstrap.cc:654] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:01.309418 18251 tablet_bootstrap.cc:492] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33: Bootstrap starting.
I20260808 13:01:01.309773 18249 tablet_bootstrap.cc:492] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa: No bootstrap required, opened a new log
I20260808 13:01:01.310096 18249 ts_tablet_manager.cc:1397] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260808 13:01:01.310523 18252 log.cc:824] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:01.313872 18252 tablet_bootstrap.cc:492] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e: No bootstrap required, opened a new log
I20260808 13:01:01.313974 18251 tablet_bootstrap.cc:654] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:01.314193 18252 ts_tablet_manager.cc:1397] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260808 13:01:01.315532 18251 log.cc:824] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:01.319190 18251 tablet_bootstrap.cc:492] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33: No bootstrap required, opened a new log
I20260808 13:01:01.319511 18251 ts_tablet_manager.cc:1397] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260808 13:01:01.322153 18249 raft_consensus.cc:348] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } } peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } }
I20260808 13:01:01.322768 18249 raft_consensus.cc:374] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:01.322880 18249 raft_consensus.cc:729] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbed5731bdde46f281e4c55af33c61fa, State: Initialized, Role: FOLLOWER
I20260808 13:01:01.322669 18252 raft_consensus.cc:348] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } } peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } }
I20260808 13:01:01.323112 18252 raft_consensus.cc:374] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:01.323235 18252 raft_consensus.cc:729] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24db0abfd3c545dd8aec45a422da099e, State: Initialized, Role: FOLLOWER
I20260808 13:01:01.323717 18249 consensus_queue.cc:260] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } } peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } }
I20260808 13:01:01.323833 18252 consensus_queue.cc:260] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } } peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } }
I20260808 13:01:01.325212 18009 heartbeater.cc:503] Master 127.24.167.62:43679 was elected leader, sending a full tablet report...
I20260808 13:01:01.325841 18249 ts_tablet_manager.cc:1428] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260808 13:01:01.327378 18218 heartbeater.cc:503] Master 127.24.167.62:43679 was elected leader, sending a full tablet report...
I20260808 13:01:01.328042 18252 ts_tablet_manager.cc:1428] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e: Time spent starting tablet: real 0.014s user 0.004s sys 0.011s
I20260808 13:01:01.331576 18251 raft_consensus.cc:348] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } } peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } }
I20260808 13:01:01.332180 18251 raft_consensus.cc:374] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:01.332321 18251 raft_consensus.cc:729] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c22d98dda44b5ebd1ac35bd84dbd33, State: Initialized, Role: FOLLOWER
I20260808 13:01:01.333123 18251 consensus_queue.cc:260] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } } peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } }
I20260808 13:01:01.335160 18251 ts_tablet_manager.cc:1428] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260808 13:01:01.349424 18256 raft_consensus.cc:482] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 13:01:01.349785 18256 raft_consensus.cc:504] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } } peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } }
I20260808 13:01:01.351646 18256 leader_election.cc:288] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36c22d98dda44b5ebd1ac35bd84dbd33 (127.24.167.1:45195), fbed5731bdde46f281e4c55af33c61fa (127.24.167.2:36913)
I20260808 13:01:01.358022 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.351969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:45195 (local address 127.24.167.3:47775)
0808 13:01:01.352190 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:01.352212 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:01.352351 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:01.352718 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:01.352723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:01.352755 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:01.353017 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:01.353033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.354259 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.354271 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:01.355937 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:01.355967 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.357091 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.357444 (+ 353us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:01.357593 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.357620 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 13:01:01.357701 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 13:01:01.357803 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 13:01:01.358215 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.352144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:47775 (local address 127.24.167.1:45195)
0808 13:01:01.352362 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:01.352370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:01.352396 (+ 26us) server_negotiation.cc:407] Connection header received
0808 13:01:01.352495 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:01.352501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:01.352575 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:01.352694 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:01.353224 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.354117 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.356130 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.356922 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.357063 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.357157 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:01.357768 (+ 611us) server_negotiation.cc:299] Negotiation successful
0808 13:01:01.357954 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 13:01:01.359134 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.352616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:41239 (local address 127.24.167.2:36913)
0808 13:01:01.352876 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:01.352884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:01.353469 (+ 585us) server_negotiation.cc:407] Connection header received
0808 13:01:01.353679 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:01.353683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:01.353742 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:01.353841 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:01.354578 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.355294 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.357199 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.358219 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.358429 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.358513 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:01.358832 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 13:01:01.358956 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260808 13:01:01.359287 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.352480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:36913 (local address 127.24.167.3:41239)
0808 13:01:01.353358 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:01.353376 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:01.353536 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:01.353857 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:01.353861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:01.353885 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:01.354355 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:01.354398 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.355435 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.355440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:01.357019 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:01.357037 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.358480 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.358486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:01.358665 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.358715 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 13:01:01.358792 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 13:01:01.358947 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":548,"threads_started":1}
I20260808 13:01:01.359481 17728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98aaf4cd2524dbc96a3435ef4619132" candidate_uuid: "24db0abfd3c545dd8aec45a422da099e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" is_pre_election: true
I20260808 13:01:01.360198 17728 raft_consensus.cc:2454] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24db0abfd3c545dd8aec45a422da099e in term 0.
I20260808 13:01:01.360289 17937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98aaf4cd2524dbc96a3435ef4619132" candidate_uuid: "24db0abfd3c545dd8aec45a422da099e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbed5731bdde46f281e4c55af33c61fa" is_pre_election: true
I20260808 13:01:01.360822 17937 raft_consensus.cc:2454] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24db0abfd3c545dd8aec45a422da099e in term 0.
I20260808 13:01:01.360991 18038 leader_election.cc:302] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24db0abfd3c545dd8aec45a422da099e, 36c22d98dda44b5ebd1ac35bd84dbd33; no voters:
I20260808 13:01:01.361447 18256 raft_consensus.cc:2781] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 13:01:01.361603 18256 raft_consensus.cc:482] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 13:01:01.361740 18256 raft_consensus.cc:3037] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:01.364130 18256 raft_consensus.cc:504] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } } peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } }
I20260808 13:01:01.364722 18256 leader_election.cc:288] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [CANDIDATE]: Term 1 election: Requested vote from peers 36c22d98dda44b5ebd1ac35bd84dbd33 (127.24.167.1:45195), fbed5731bdde46f281e4c55af33c61fa (127.24.167.2:36913)
I20260808 13:01:01.365248 17728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98aaf4cd2524dbc96a3435ef4619132" candidate_uuid: "24db0abfd3c545dd8aec45a422da099e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33"
I20260808 13:01:01.365454 17728 raft_consensus.cc:3037] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:01.365429 17937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98aaf4cd2524dbc96a3435ef4619132" candidate_uuid: "24db0abfd3c545dd8aec45a422da099e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbed5731bdde46f281e4c55af33c61fa"
I20260808 13:01:01.365658 17937 raft_consensus.cc:3037] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:01.367893 17728 raft_consensus.cc:2454] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24db0abfd3c545dd8aec45a422da099e in term 1.
I20260808 13:01:01.368142 17937 raft_consensus.cc:2454] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24db0abfd3c545dd8aec45a422da099e in term 1.
I20260808 13:01:01.368341 18038 leader_election.cc:302] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24db0abfd3c545dd8aec45a422da099e, 36c22d98dda44b5ebd1ac35bd84dbd33; no voters:
I20260808 13:01:01.368675 18256 raft_consensus.cc:2781] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:01:01.369174 18256 raft_consensus.cc:686] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [term 1 LEADER]: Becoming Leader. State: Replica: 24db0abfd3c545dd8aec45a422da099e, State: Running, Role: LEADER
I20260808 13:01:01.369695 18256 consensus_queue.cc:237] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } } peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } }
I20260808 13:01:01.374763 17510 catalog_manager.cc:5495] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e reported cstate change: term changed from 0 to 1, leader changed from <none> to 24db0abfd3c545dd8aec45a422da099e (127.24.167.3). New cstate: current_term: 1 leader_uuid: "24db0abfd3c545dd8aec45a422da099e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24db0abfd3c545dd8aec45a422da099e" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 40393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 } health_report { overall_health: UNKNOWN } } }
I20260808 13:01:01.401984 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.396210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.24.167.3:40393)
0808 13:01:01.396438 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:01.396442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:01.396462 (+ 20us) server_negotiation.cc:407] Connection header received
0808 13:01:01.396720 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:01.396725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:01.396815 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:01.396965 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:01:01.397337 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.398157 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.399638 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.400091 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.400208 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.400724 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:01:01.401296 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:01:01.401458 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:01.401622 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 13:01:01.401763 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260808 13:01:01.410728 18010 tablet.cc:2369] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:01:01.411478 17728 raft_consensus.cc:1261] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 1 FOLLOWER]: Refusing update from remote peer 24db0abfd3c545dd8aec45a422da099e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:01:01.411477 17937 raft_consensus.cc:1261] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa [term 1 FOLLOWER]: Refusing update from remote peer 24db0abfd3c545dd8aec45a422da099e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:01:01.412751 18261 consensus_queue.cc:1035] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbed5731bdde46f281e4c55af33c61fa" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 36913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:01:01.413173 18256 consensus_queue.cc:1035] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [LEADER]: Connected to new peer: Peer: permanent_uuid: "36c22d98dda44b5ebd1ac35bd84dbd33" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 13:01:01.414255 17800 tablet.cc:2369] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 13:01:01.423207 18219 tablet.cc:2369] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:01:01.441840 17511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:01:01.442186 17511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:01:01.446175 17511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Sending DeleteTablet for 3 replicas of tablet e98aaf4cd2524dbc96a3435ef4619132
I20260808 13:01:01.447543 18114 tablet_service.cc:1514] Processing DeleteTablet for tablet e98aaf4cd2524dbc96a3435ef4619132 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:01 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:44206
I20260808 13:01:01.447643 17695 tablet_service.cc:1514] Processing DeleteTablet for tablet e98aaf4cd2524dbc96a3435ef4619132 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:01 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:43344
I20260808 13:01:01.447810 17903 tablet_service.cc:1514] Processing DeleteTablet for tablet e98aaf4cd2524dbc96a3435ef4619132 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:01 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:52698
I20260808 13:01:01.448551 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 17564
I20260808 13:01:01.461180 18278 tablet_replica.cc:330] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e: stopping tablet replica
I20260808 13:01:01.461879 18278 raft_consensus.cc:2229] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [term 1 LEADER]: Raft consensus shutting down.
I20260808 13:01:01.462920 18278 raft_consensus.cc:2258] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:01.464131 18279 tablet_replica.cc:330] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33: stopping tablet replica
I20260808 13:01:01.464787 18279 raft_consensus.cc:2229] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:01:01.465534 18279 raft_consensus.cc:2258] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:01.466603 18280 tablet_replica.cc:330] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa: stopping tablet replica
I20260808 13:01:01.467056 18280 raft_consensus.cc:2229] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:01:01.467573 18280 raft_consensus.cc:2258] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:01.469879 18279 ts_tablet_manager.cc:1905] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:01.469880 18278 ts_tablet_manager.cc:1905] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:01.470309 18280 ts_tablet_manager.cc:1905] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:01.477376 18279 ts_tablet_manager.cc:1918] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:01.477595 18279 log.cc:1196] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/e98aaf4cd2524dbc96a3435ef4619132
I20260808 13:01:01.478201 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.449535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.24.167.1:45195)
0808 13:01:01.450114 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:01.450121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:01.450142 (+ 21us) server_negotiation.cc:407] Connection header received
0808 13:01:01.450203 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:01.450209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:01.450320 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:01.450511 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:01.452578 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.453798 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.456078 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.456766 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.456913 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.463982 (+ 7069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:01.464003 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:01.464023 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:01.464080 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:01.468507 (+ 4427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:01.470491 (+ 1984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:01.470503 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:01.470506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:01.470713 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:01.475938 (+ 5225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:01.475945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:01.475949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:01.476630 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:01.477053 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:01.477446 (+ 393us) server_negotiation.cc:299] Negotiation successful
0808 13:01:01.477745 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446}
I20260808 13:01:01.478281 18279 ts_tablet_manager.cc:1939] T e98aaf4cd2524dbc96a3435ef4619132 P 36c22d98dda44b5ebd1ac35bd84dbd33: Deleting consensus metadata
I20260808 13:01:01.478994 17791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:01.479720 18278 ts_tablet_manager.cc:1918] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:01.479961 18278 log.cc:1196] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/e98aaf4cd2524dbc96a3435ef4619132
I20260808 13:01:01.480518 18278 ts_tablet_manager.cc:1939] T e98aaf4cd2524dbc96a3435ef4619132 P 24db0abfd3c545dd8aec45a422da099e: Deleting consensus metadata
I20260808 13:01:01.482492 17490 catalog_manager.cc:4841] TS 24db0abfd3c545dd8aec45a422da099e (127.24.167.3:40393): tablet e98aaf4cd2524dbc96a3435ef4619132 (table test-table [id=5f97fb61e5fd46c986e5c203f9741eb1]) successfully deleted
I20260808 13:01:01.485491 18280 ts_tablet_manager.cc:1918] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:01.485728 18280 log.cc:1196] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/e98aaf4cd2524dbc96a3435ef4619132
I20260808 13:01:01.486277 18280 ts_tablet_manager.cc:1939] T e98aaf4cd2524dbc96a3435ef4619132 P fbed5731bdde46f281e4c55af33c61fa: Deleting consensus metadata
I20260808 13:01:01.488229 17492 catalog_manager.cc:4841] TS fbed5731bdde46f281e4c55af33c61fa (127.24.167.2:36913): tablet e98aaf4cd2524dbc96a3435ef4619132 (table test-table [id=5f97fb61e5fd46c986e5c203f9741eb1]) successfully deleted
I20260808 13:01:01.575484 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 17564
I20260808 13:01:01.575495 17492 catalog_manager.cc:4841] TS 36c22d98dda44b5ebd1ac35bd84dbd33 (127.24.167.1:45195): tablet e98aaf4cd2524dbc96a3435ef4619132 (table test-table [id=5f97fb61e5fd46c986e5c203f9741eb1]) successfully deleted
I20260808 13:01:01.606730 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 17803
I20260808 13:01:01.626067 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.607668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59698 (local address 127.24.167.2:36913)
0808 13:01:01.607884 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:01.607890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:01.607914 (+ 24us) server_negotiation.cc:407] Connection header received
0808 13:01:01.608090 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:01.608096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:01.608178 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:01.608289 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:01.610108 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.611474 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.613522 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.614190 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.614326 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.617996 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:01.618019 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:01.618042 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:01.618107 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:01.622611 (+ 4504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:01.623538 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:01.623550 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:01.623554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:01.623737 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:01.624150 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:01.624155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:01.624159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:01.624713 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:01.625064 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:01.625388 (+ 324us) server_negotiation.cc:299] Negotiation successful
0808 13:01:01.625647 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 13:01:01.626906 18003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:01.734489 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 17803
I20260808 13:01:01.763684 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 18013
I20260808 13:01:01.782203 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:01.764642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.24.167.3:40393)
0808 13:01:01.764838 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:01.764843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:01.764936 (+ 93us) server_negotiation.cc:407] Connection header received
0808 13:01:01.765136 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:01.765140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:01.765218 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:01.765307 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:01.767421 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.768256 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.770324 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:01.770873 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:01.770977 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:01.774615 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:01.774629 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:01.774647 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:01.774694 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:01.778724 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:01.779748 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:01.779756 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:01.779759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:01.779945 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:01.780496 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:01.780500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:01.780503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:01.780894 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:01.781130 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:01.781601 (+ 471us) server_negotiation.cc:299] Negotiation successful
0808 13:01:01.781797 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 13:01:01.783228 18210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:01.910274 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 18013
I20260808 13:01:01.933423 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 17463
I20260808 13:01:01.934777 17556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:02.051273 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 17463
I20260808 13:01:22.084364 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:43679
--webserver_interface=127.24.167.62
--webserver_port=38055
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:43679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:01:22.451148 18288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:22.451608 18288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:22.451792 18288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:22.465554 18288 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 13:01:22.465713 18288 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 13:01:22.465776 18288 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 13:01:22.465859 18288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:01:22.465955 18288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:22.466058 18288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:22.466132 18288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:01:22.466219 18288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:01:22.485531 18288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:43679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:43679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=38055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:22.487486 18288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:22.489373 18288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:22.503455 18296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:22.503500 18297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:22.504665 18299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:22.504630 18288 server_base.cc:1034] running on GCE node
I20260808 13:01:22.505724 18288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:22.507632 18288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:22.508916 18288 hybrid_clock.cc:648] HybridClock initialized: now 1786194082508814 us; error 54 us; skew 500 ppm
I20260808 13:01:22.509383 18288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:22 dist-test-slave-tk9z krb5kdc[16570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:22.517966 18288 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:01:22.520191 18288 webserver.cc:460] Webserver started at http://127.24.167.62:38055/ using document root <none> and password file <none>
I20260808 13:01:22.521312 18288 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:22.521459 18288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:22.528898 18288 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 13:01:22.531782 18309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:22.533095 18288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:01:22.533291 18288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "d7335399a36e424da32d6a3b5c8c32a4"
format_stamp: "Formatted at 2026-08-08 13:00:36 on dist-test-slave-tk9z"
server_key: "d7f1c3616429fc452efd402e0ece60a7"
server_key_iv: "76ecc67eb9f5b27fac1edc61f9466a31"
server_key_version: "encryptionkey@0"
I20260808 13:01:22.534281 18288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:22.570881 18288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:22.575453 18288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:22.575794 18288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:22.596139 18288 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:43679
I20260808 13:01:22.596181 18398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:43679 every 8 connection(s)
I20260808 13:01:22.597703 18288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:01:22.604920 18401 sys_catalog.cc:263] Verifying existing consensus state
I20260808 13:01:22.606113 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 18288
I20260808 13:01:22.607223 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:45195
--local_ip_for_outbound_sockets=127.24.167.1
--tserver_master_addrs=127.24.167.62:43679
--webserver_port=45813
--webserver_interface=127.24.167.1
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 13:01:22.609879 18401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Bootstrap starting.
I20260808 13:01:22.633896 18401 log.cc:824] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:22.652020 18401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 13:01:22.652709 18401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Bootstrap complete.
I20260808 13:01:22.662569 18401 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } }
I20260808 13:01:22.663553 18401 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7335399a36e424da32d6a3b5c8c32a4, State: Initialized, Role: FOLLOWER
I20260808 13:01:22.664183 18401 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } }
I20260808 13:01:22.664444 18401 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:01:22.664575 18401 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:01:22.664760 18401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 13:01:22.668866 18401 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } }
I20260808 13:01:22.669421 18401 leader_election.cc:302] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7335399a36e424da32d6a3b5c8c32a4; no voters:
I20260808 13:01:22.670156 18401 leader_election.cc:288] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 13:01:22.670333 18412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 13:01:22.671609 18412 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [term 3 LEADER]: Becoming Leader. State: Replica: d7335399a36e424da32d6a3b5c8c32a4, State: Running, Role: LEADER
I20260808 13:01:22.672326 18412 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } }
I20260808 13:01:22.672591 18401 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:01:22.675112 18413 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d7335399a36e424da32d6a3b5c8c32a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } } }
I20260808 13:01:22.675616 18413 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [sys.catalog]: This master's current role is: LEADER
I20260808 13:01:22.679594 18414 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7335399a36e424da32d6a3b5c8c32a4. Latest consensus state: current_term: 3 leader_uuid: "d7335399a36e424da32d6a3b5c8c32a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7335399a36e424da32d6a3b5c8c32a4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 43679 } } }
I20260808 13:01:22.679950 18414 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4 [sys.catalog]: This master's current role is: LEADER
I20260808 13:01:22.681746 18421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:01:22.692593 18421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:01:22.694546 18421 catalog_manager.cc:1259] Loaded cluster ID: 80bbd6c7076446359f9faf247c4ce482
I20260808 13:01:22.694684 18421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:01:22.699846 18421 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:01:22.700143 18438 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 13:01:22.701483 18421 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Loaded TSK: 0
I20260808 13:01:22.701629 18421 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d7335399a36e424da32d6a3b5c8c32a4: Loaded TSK: 1
F20260808 13:01:22.702736 18421 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0a7740c9ad google::LogMessage::Fail() at ??:0
@ 0x7f0a7740f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f0a7740c4de google::LogMessage::Flush() at ??:0
@ 0x7f0a7740da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0a7b1707d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0a7b15f73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f0a8ce3b1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f0a8ce3a783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f0a8cec62a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f0a8cec60c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f0a8cf105e9 std::function<>::operator()() at ??:0
@ 0x7f0a8ce3ecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0a8ce3cc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0a8cec4610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0a8cec4439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0a7a058345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0a7a05b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0a7a05b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0a7a036eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0a7d3106db start_thread at ??:0
@ 0x7f0a762a661f clone at ??:0
W20260808 13:01:23.090229 18405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:23.090638 18405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:23.090762 18405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:23.098240 18405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:23.098415 18405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:23.098563 18405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 13:01:23.108816 18405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:45195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=45813
--tserver_master_addrs=127.24.167.62:43679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:23.110146 18405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:23.111653 18405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:23.125306 18473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:23.125264 18474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:23.125362 18481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:23.126238 18405 server_base.cc:1034] running on GCE node
I20260808 13:01:23.126984 18405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:23.127974 18405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:23.129176 18405 hybrid_clock.cc:648] HybridClock initialized: now 1786194083129115 us; error 59 us; skew 500 ppm
I20260808 13:01:23.129509 18405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:23 dist-test-slave-tk9z krb5kdc[16570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194083, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:23.132800 18405 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 13:01:23.134438 18405 webserver.cc:460] Webserver started at http://127.24.167.1:45813/ using document root <none> and password file <none>
I20260808 13:01:23.135171 18405 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:23.135317 18405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:23.140945 18405 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 13:01:23.143842 18490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:23.145274 18405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:01:23.145460 18405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "36c22d98dda44b5ebd1ac35bd84dbd33"
format_stamp: "Formatted at 2026-08-08 13:00:37 on dist-test-slave-tk9z"
server_key: "43c0cd2965b824e8d3055229ea7f594c"
server_key_iv: "0bc28cd3f579af634199aa24a844ffa6"
server_key_version: "encryptionkey@0"
I20260808 13:01:23.146593 18405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:23.174631 18405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:23.178931 18405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:23.179222 18405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:23.180315 18405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:23.181756 18405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:23.181870 18405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:23.182009 18405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:23.182093 18405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:23.217842 18405 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:45195
I20260808 13:01:23.217967 18664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:45195 every 8 connection(s)
I20260808 13:01:23.219514 18405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 13:01:23.224761 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:23.222763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:51713)
0808 13:01:23.223578 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:23.223838 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":336,"threads_started":1}
I20260808 13:01:23.225193 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 18405
W20260808 13:01:23.225574 18665 heartbeater.cc:644] Failed to heartbeat to 127.24.167.62:43679 (0 consecutive failures): Network error: Failed to ping master at 127.24.167.62:43679: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
I20260808 13:01:23.225920 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:36913
--local_ip_for_outbound_sockets=127.24.167.2
--tserver_master_addrs=127.24.167.62:43679
--webserver_port=34853
--webserver_interface=127.24.167.2
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 13:01:23.226533 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:23.226119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:52115)
0808 13:01:23.226317 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:23.226426 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260808 13:01:23.227433 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:23.227088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:45671)
0808 13:01:23.227230 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:23.227315 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 13:01:23.227682 18665 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 13:01:23.587634 18668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:23.587924 18668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:23.588018 18668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:23.595642 18668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:23.595763 18668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:23.595878 18668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 13:01:23.606235 18668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:36913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=34853
--tserver_master_addrs=127.24.167.62:43679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:23.607586 18668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:23.608901 18668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:23.622674 18674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:23.622705 18675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:23.622764 18680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:23.623588 18668 server_base.cc:1034] running on GCE node
I20260808 13:01:23.624231 18668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:23.625277 18668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:23.626510 18668 hybrid_clock.cc:648] HybridClock initialized: now 1786194083626437 us; error 58 us; skew 500 ppm
I20260808 13:01:23.626843 18668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:23 dist-test-slave-tk9z krb5kdc[16570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194083, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:23.630023 18668 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 13:01:23.631624 18668 webserver.cc:460] Webserver started at http://127.24.167.2:34853/ using document root <none> and password file <none>
I20260808 13:01:23.632313 18668 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:23.632463 18668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:23.637725 18668 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 13:01:23.640508 18694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:23.641830 18668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 13:01:23.642014 18668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "fbed5731bdde46f281e4c55af33c61fa"
format_stamp: "Formatted at 2026-08-08 13:00:37 on dist-test-slave-tk9z"
server_key: "04ed2b600a808f6d2dcdfd2b8d3f736f"
server_key_iv: "6cebe99f537bcea6bc68a0daf5f6932e"
server_key_version: "encryptionkey@0"
I20260808 13:01:23.643106 18668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:23.659358 18668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:23.663853 18668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:23.664168 18668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:23.665237 18668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:23.666759 18668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:23.666894 18668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:23.667021 18668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:23.667117 18668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:23.703398 18668 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:36913
I20260808 13:01:23.703495 18868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:36913 every 8 connection(s)
I20260808 13:01:23.704977 18668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 13:01:23.707126 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 18668
I20260808 13:01:23.707862 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:40393
--local_ip_for_outbound_sockets=127.24.167.3
--tserver_master_addrs=127.24.167.62:43679
--webserver_port=44311
--webserver_interface=127.24.167.3
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 13:01:23.710745 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:23.708536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:45593)
0808 13:01:23.709446 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:23.709744 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":395,"threads_started":1}
W20260808 13:01:23.711472 18869 heartbeater.cc:644] Failed to heartbeat to 127.24.167.62:43679 (0 consecutive failures): Network error: Failed to ping master at 127.24.167.62:43679: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
I20260808 13:01:23.712500 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:23.712092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:45047)
0808 13:01:23.712274 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:23.712364 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 13:01:23.713533 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:23.713140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:59941)
0808 13:01:23.713297 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:23.713389 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 13:01:23.713840 18869 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 13:01:24.060058 18873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:24.060375 18873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:24.060477 18873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:24.067943 18873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:24.068063 18873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:24.068177 18873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 13:01:24.078153 18873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:40393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=44311
--tserver_master_addrs=127.24.167.62:43679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:24.079545 18873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:24.080858 18873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:24.095007 18881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:24.094928 18882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:24.094983 18885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:24.095878 18873 server_base.cc:1034] running on GCE node
I20260808 13:01:24.096537 18873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:24.097520 18873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:24.098754 18873 hybrid_clock.cc:648] HybridClock initialized: now 1786194084098679 us; error 59 us; skew 500 ppm
I20260808 13:01:24.099089 18873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:24 dist-test-slave-tk9z krb5kdc[16570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194084, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:24.102528 18873 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 13:01:24.104161 18873 webserver.cc:460] Webserver started at http://127.24.167.3:44311/ using document root <none> and password file <none>
I20260808 13:01:24.104944 18873 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:24.105098 18873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:24.110975 18873 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 13:01:24.113842 18895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:24.115300 18873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:01:24.115536 18873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "24db0abfd3c545dd8aec45a422da099e"
format_stamp: "Formatted at 2026-08-08 13:00:38 on dist-test-slave-tk9z"
server_key: "8b3288ea8ec68d1d318ed0e6fc6f1b47"
server_key_iv: "cf026565e609df9f308435af70268ab0"
server_key_version: "encryptionkey@0"
I20260808 13:01:24.116617 18873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:24.151821 18873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:24.155936 18873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:24.156219 18873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:24.157239 18873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:24.158571 18873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:24.158700 18873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:24.158824 18873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:24.158914 18873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:24.192510 18873 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:40393
I20260808 13:01:24.192601 19066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:40393 every 8 connection(s)
I20260808 13:01:24.194103 18873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 13:01:24.198928 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:24.197104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:49919)
0808 13:01:24.197813 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:24.198056 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":268,"threads_started":1}
W20260808 13:01:24.199676 19067 heartbeater.cc:644] Failed to heartbeat to 127.24.167.62:43679 (0 consecutive failures): Network error: Failed to ping master at 127.24.167.62:43679: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
I20260808 13:01:24.200162 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 18873
I20260808 13:01:24.200584 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:24.200183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:58909)
0808 13:01:24.200351 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:24.200439 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 13:01:24.201503 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:24.201163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:34065)
0808 13:01:24.201306 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:24.201383 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 13:01:24.201807 19067 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 13:01:24.229094 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:24.228253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:54645)
0808 13:01:24.228734 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:24.228833 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":229,"threads_started":1}
I20260808 13:01:24.715350 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:24.714443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:54523)
0808 13:01:24.714974 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:24.715107 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":235,"threads_started":1}
I20260808 13:01:25.203249 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:25.202317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:58599)
0808 13:01:25.202925 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:25.203020 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":279,"threads_started":1}
I20260808 13:01:25.231011 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:25.230118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:46895)
0808 13:01:25.230694 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:25.230822 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":323,"threads_started":1}
I20260808 13:01:25.717222 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:25.716419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:48815)
0808 13:01:25.716959 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:25.717055 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":241,"threads_started":1}
I20260808 13:01:26.204996 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:26.204263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:42419)
0808 13:01:26.204727 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:26.204819 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":224,"threads_started":1}
I20260808 13:01:26.232708 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:26.231938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:40051)
0808 13:01:26.232420 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:26.232511 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":247,"threads_started":1}
I20260808 13:01:26.718904 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:26.718119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:43265)
0808 13:01:26.718633 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:26.718719 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":278,"threads_started":1}
I20260808 13:01:27.206885 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:27.206047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:54387)
0808 13:01:27.206607 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:27.206700 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":246,"threads_started":1}
I20260808 13:01:27.234516 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:27.233698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:40681)
0808 13:01:27.234167 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:27.234253 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":240,"threads_started":1}
I20260808 13:01:27.720830 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:27.719943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:45875)
0808 13:01:27.720447 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:27.720628 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":266,"threads_started":1}
I20260808 13:01:28.208577 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:28.207824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:57581)
0808 13:01:28.208315 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:28.208403 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":259,"threads_started":1}
I20260808 13:01:28.236316 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:28.235560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:43739)
0808 13:01:28.236051 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:28.236142 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":242,"threads_started":1}
I20260808 13:01:28.722805 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:28.721924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:55785)
0808 13:01:28.722480 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:28.722610 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":246,"threads_started":1}
I20260808 13:01:29.210525 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:29.209589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:56211)
0808 13:01:29.210155 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:29.210280 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":272,"threads_started":1}
I20260808 13:01:29.238052 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:29.237278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:41211)
0808 13:01:29.237763 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:29.237863 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":243,"threads_started":1}
I20260808 13:01:29.724663 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:29.723842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:41081)
0808 13:01:29.724360 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:29.724484 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":259,"threads_started":1}
I20260808 13:01:30.212221 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:30.211466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:39359)
0808 13:01:30.211961 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:30.212059 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":280,"threads_started":1}
I20260808 13:01:30.239675 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:30.238885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:51867)
0808 13:01:30.239387 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:30.239468 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":224,"threads_started":1}
I20260808 13:01:30.726557 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:30.725661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:59187)
0808 13:01:30.726174 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:30.726271 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":226,"threads_started":1}
I20260808 13:01:31.214005 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:31.213188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:48517)
0808 13:01:31.213695 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:31.213781 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":233,"threads_started":1}
I20260808 13:01:31.241367 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:31.240646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:58015)
0808 13:01:31.241116 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:31.241203 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
I20260808 13:01:31.728366 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:31.727619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:37775)
0808 13:01:31.728111 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:31.728198 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":227,"threads_started":1}
I20260808 13:01:32.215788 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:32.215063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:43461)
0808 13:01:32.215521 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:32.215603 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":260,"threads_started":1}
I20260808 13:01:32.243119 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:32.242364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:35847)
0808 13:01:32.242861 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:32.242941 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":216,"threads_started":1}
I20260808 13:01:32.730120 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:32.729308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:56451)
0808 13:01:32.729826 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:32.729927 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":257,"threads_started":1}
I20260808 13:01:33.217594 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:33.216738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:35163)
0808 13:01:33.217298 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:33.217416 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":292,"threads_started":1}
I20260808 13:01:33.244784 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:33.244061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:47017)
0808 13:01:33.244537 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:33.244622 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":256,"threads_started":1}
I20260808 13:01:33.731953 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:33.731186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:60605)
0808 13:01:33.731667 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:33.731755 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
I20260808 13:01:34.219703 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:34.218806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:43745)
0808 13:01:34.219399 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:34.219536 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":346,"threads_started":1}
I20260808 13:01:34.246703 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:34.245807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:36775)
0808 13:01:34.246336 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:34.246511 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":296,"threads_started":1}
I20260808 13:01:34.733681 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:34.732931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:59505)
0808 13:01:34.733394 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:34.733487 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":217,"threads_started":1}
I20260808 13:01:35.221674 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:35.220754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:57255)
0808 13:01:35.221294 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:35.221394 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":294,"threads_started":1}
I20260808 13:01:35.248308 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:35.247595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:43411)
0808 13:01:35.248064 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:35.248148 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":244,"threads_started":1}
I20260808 13:01:35.735505 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:35.734741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:39711)
0808 13:01:35.735248 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:35.735338 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":271,"threads_started":1}
I20260808 13:01:36.223387 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:36.222718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:43871)
0808 13:01:36.223152 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:36.223233 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":228,"threads_started":1}
I20260808 13:01:36.250281 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:36.249403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:58769)
0808 13:01:36.249931 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:36.250060 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":259,"threads_started":1}
I20260808 13:01:36.737401 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:36.736539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:36159)
0808 13:01:36.737062 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:36.737205 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":262,"threads_started":1}
I20260808 13:01:37.225171 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:37.224327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:34257)
0808 13:01:37.224861 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:37.224970 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":264,"threads_started":1}
I20260808 13:01:37.252024 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:37.251266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:41029)
0808 13:01:37.251754 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:37.251841 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":274,"threads_started":1}
I20260808 13:01:37.739192 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:37.738449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:34603)
0808 13:01:37.738901 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:37.739005 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":219,"threads_started":1}
I20260808 13:01:38.226999 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:38.226124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:44647)
0808 13:01:38.226704 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:38.226813 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":297,"threads_started":1}
I20260808 13:01:38.254046 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:38.253173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:53647)
0808 13:01:38.253743 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:38.253854 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":335,"threads_started":1}
I20260808 13:01:38.741184 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:38.740479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.2:36797)
0808 13:01:38.740911 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:38.741003 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":211,"threads_started":1}
I20260808 13:01:39.208837 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 18405
I20260808 13:01:39.229202 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:39.228237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.3:40989)
0808 13:01:39.228837 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:39.228958 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":298,"threads_started":1}
I20260808 13:01:39.234866 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:39.210448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.24.167.1:45195)
0808 13:01:39.211000 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:39.211015 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:39.211054 (+ 39us) server_negotiation.cc:407] Connection header received
0808 13:01:39.212650 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:39.212660 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:39.213090 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:39.213439 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:39.215083 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:39.216465 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:39.217793 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:39.218570 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:39.218776 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:39.227163 (+ 8387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:39.227198 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:39.227227 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:39.227300 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:39.231190 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:39.232015 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:39.232022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:39.232030 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:39.232206 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:39.232651 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:39.232655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:39.232657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:39.233237 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:39.233487 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:39.234074 (+ 587us) server_negotiation.cc:299] Negotiation successful
0808 13:01:39.234538 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":233,"threads_started":1}
I20260808 13:01:39.236409 18655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:39.323328 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 18405
I20260808 13:01:39.323433 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:39.322091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:43679 (local address 127.24.167.1:43289)
0808 13:01:39.322812 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:39.323095 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.62:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":372,"threads_started":1}
I20260808 13:01:39.353215 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 18668
I20260808 13:01:39.371750 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:39.354283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36998 (local address 127.24.167.2:36913)
0808 13:01:39.355015 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:39.355030 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:39.355071 (+ 41us) server_negotiation.cc:407] Connection header received
0808 13:01:39.356665 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:39.356676 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:39.357119 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:39.357415 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:39.358994 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:39.360282 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:39.361533 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:39.362203 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:39.362361 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:39.365350 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:39.365358 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:39.365373 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:39.365413 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:39.368434 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:39.369195 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:39.369202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:39.369209 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:39.369382 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:39.369809 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:39.369812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:39.369814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:39.370362 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:39.370708 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:39.371100 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 13:01:39.371433 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":367,"threads_started":1}
I20260808 13:01:39.373150 18863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:39.458216 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 18668
I20260808 13:01:39.483999 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 18873
I20260808 13:01:39.502483 19163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:39.485165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40502 (local address 127.24.167.3:40393)
0808 13:01:39.485808 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:39.485822 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:39.485866 (+ 44us) server_negotiation.cc:407] Connection header received
0808 13:01:39.487598 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:39.487608 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:39.488027 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:39.488315 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:39.489643 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:39.491142 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:39.492428 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:39.493114 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:39.493255 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:39.496052 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:39.496060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:39.496078 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:39.496126 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:39.499172 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:39.499889 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:39.499896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:39.499903 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:39.500077 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:39.500418 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:39.500421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:39.500423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:39.501006 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:39.501231 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:39.501602 (+ 371us) server_negotiation.cc:299] Negotiation successful
0808 13:01:39.502037 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":316,"threads_started":1}
I20260808 13:01:39.503882 19060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:39.587860 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 18873
2026-08-08T13:01:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65329 ms)
[----------] 30 tests from SecurityITest (315491 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:01:39 dist-test-slave-tk9z krb5kdc[19167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:01:39 dist-test-slave-tk9z krb5kdc[19167](info): set up 2 sockets
Aug 08 13:01:39 dist-test-slave-tk9z krb5kdc[19167](info): commencing operation
krb5kdc: starting...
W20260808 13:01:40.928224 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.001s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:01:41 dist-test-slave-tk9z krb5kdc[19167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:01:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:01:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 13:01:41.254544 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:42185
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:42185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:01:41.607178 19183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:41.607484 19183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:41.607645 19183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:41.615005 19183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:01:41.615113 19183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:41.615227 19183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:41.615330 19183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:01:41.615412 19183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:01:41.625460 19183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:42185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:42185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:41.626860 19183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:41.628202 19183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:41.640733 19193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:41.640785 19195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:41.640734 19192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:41.641513 19183 server_base.cc:1034] running on GCE node
I20260808 13:01:41.642423 19183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:41.643950 19183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:41.645212 19183 hybrid_clock.cc:648] HybridClock initialized: now 1786194101645169 us; error 49 us; skew 500 ppm
I20260808 13:01:41.645634 19183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:41 dist-test-slave-tk9z krb5kdc[19167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194101, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:41.648939 19183 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:01:41.650586 19183 webserver.cc:460] Webserver started at http://127.24.167.62:36009/ using document root <none> and password file <none>
I20260808 13:01:41.651319 19183 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:41.651480 19183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:41.651824 19183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:41.655058 19183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "d7b86038097a41928d4fd3ad4515ea88"
format_stamp: "Formatted at 2026-08-08 13:01:41 on dist-test-slave-tk9z"
server_key: "a77ea32905abfec105f13f2157aa307d"
server_key_iv: "268a5bab61da689c5c342ce72c95aefb"
server_key_version: "encryptionkey@0"
I20260808 13:01:41.655854 19183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "d7b86038097a41928d4fd3ad4515ea88"
format_stamp: "Formatted at 2026-08-08 13:01:41 on dist-test-slave-tk9z"
server_key: "a77ea32905abfec105f13f2157aa307d"
server_key_iv: "268a5bab61da689c5c342ce72c95aefb"
server_key_version: "encryptionkey@0"
I20260808 13:01:41.661403 19183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260808 13:01:41.665110 19205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:41.666508 19183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:01:41.666702 19183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "d7b86038097a41928d4fd3ad4515ea88"
format_stamp: "Formatted at 2026-08-08 13:01:41 on dist-test-slave-tk9z"
server_key: "a77ea32905abfec105f13f2157aa307d"
server_key_iv: "268a5bab61da689c5c342ce72c95aefb"
server_key_version: "encryptionkey@0"
I20260808 13:01:41.666916 19183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:41.681097 19183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:41.685304 19183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:41.685611 19183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:41.705011 19183 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:42185
I20260808 13:01:41.705041 19282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:42185 every 8 connection(s)
I20260808 13:01:41.706504 19183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:01:41.710389 19284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:41.714054 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 19183
I20260808 13:01:41.714296 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:01:41.715193 25244 external_mini_cluster.cc:1428] Setting key 8d5489032f81d4eb2fdb150b7d801a57
I20260808 13:01:41.724493 19284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88: Bootstrap starting.
I20260808 13:01:41.728554 19284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:41.729974 19284 log.cc:824] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88: Log is configured to *not* fsync() on all Append() calls
Aug 08 13:01:41 dist-test-slave-tk9z krb5kdc[19167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:01:41.733266 19284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88: No bootstrap required, opened a new log
I20260808 13:01:41.740409 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:41.717341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.24.167.62:42185)
0808 13:01:41.718111 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:41.718123 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:41.718163 (+ 40us) server_negotiation.cc:407] Connection header received
0808 13:01:41.719953 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:41.719977 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:41.720389 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:41.720737 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:41.722757 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:41.724090 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:41.726018 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:41.727005 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:41.727171 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:41.732679 (+ 5508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:41.732687 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:41.732701 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:41.732737 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:41.736112 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:41.736898 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:41.736905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:41.736911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:41.737105 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:41.737519 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:41.737534 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:41.737537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:41.738108 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:41.738458 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:41.738955 (+ 497us) server_negotiation.cc:299] Negotiation successful
0808 13:01:41.739390 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":342,"threads_started":1}
I20260808 13:01:41.741658 19284 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b86038097a41928d4fd3ad4515ea88" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42185 } }
I20260808 13:01:41.742075 19284 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:41.742177 19284 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b86038097a41928d4fd3ad4515ea88, State: Initialized, Role: FOLLOWER
I20260808 13:01:41.742844 19284 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b86038097a41928d4fd3ad4515ea88" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42185 } }
I20260808 13:01:41.743103 19284 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:01:41.743273 19284 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:01:41.743467 19284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:41.745740 19284 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b86038097a41928d4fd3ad4515ea88" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42185 } }
I20260808 13:01:41.746238 19284 leader_election.cc:302] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7b86038097a41928d4fd3ad4515ea88; no voters:
I20260808 13:01:41.746881 19284 leader_election.cc:288] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:01:41.747022 19292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:01:41.747807 19292 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [term 1 LEADER]: Becoming Leader. State: Replica: d7b86038097a41928d4fd3ad4515ea88, State: Running, Role: LEADER
I20260808 13:01:41.748400 19292 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b86038097a41928d4fd3ad4515ea88" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42185 } }
I20260808 13:01:41.749223 19284 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:01:41.751955 19297 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7b86038097a41928d4fd3ad4515ea88. Latest consensus state: current_term: 1 leader_uuid: "d7b86038097a41928d4fd3ad4515ea88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b86038097a41928d4fd3ad4515ea88" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42185 } } }
I20260808 13:01:41.752244 19295 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7b86038097a41928d4fd3ad4515ea88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b86038097a41928d4fd3ad4515ea88" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42185 } } }
I20260808 13:01:41.752521 19297 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [sys.catalog]: This master's current role is: LEADER
I20260808 13:01:41.752521 19295 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88 [sys.catalog]: This master's current role is: LEADER
I20260808 13:01:41.755432 19309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:01:41.760927 19309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:01:41.771059 19309 catalog_manager.cc:1347] Generated new cluster ID: 5202b8891c64416c93e088eb2b489c58
I20260808 13:01:41.771209 19309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:01:41.783738 19309 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:01:41.785177 19309 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:01:41.792913 19309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88: Generated new TSK 0
I20260808 13:01:41.793792 19309 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 13:01:41.863561 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42185
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:01:42.218638 19326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:42.219017 19326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:42.219208 19326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:42.226856 19326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:42.227020 19326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:42.227219 19326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 13:01:42.237717 19326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:42.239240 19326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:42.240705 19326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:42.253913 19333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:42.253800 19337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:42.253805 19334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:42.260097 19326 server_base.cc:1034] running on GCE node
I20260808 13:01:42.260715 19326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:42.261718 19326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:42.262929 19326 hybrid_clock.cc:648] HybridClock initialized: now 1786194102262875 us; error 40 us; skew 500 ppm
I20260808 13:01:42.263244 19326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:42 dist-test-slave-tk9z krb5kdc[19167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194102, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:42.266682 19326 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 13:01:42.268299 19326 webserver.cc:460] Webserver started at http://127.24.167.1:42365/ using document root <none> and password file <none>
I20260808 13:01:42.269042 19326 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:42.269191 19326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:42.269518 19326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:42.272804 19326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "fde2a28d8b7840c89355dd65fa1c09de"
format_stamp: "Formatted at 2026-08-08 13:01:42 on dist-test-slave-tk9z"
server_key: "83abfa8d9ef3d5196b72cd7591f23a37"
server_key_iv: "0382cf36a87402a82cf94bf82f799ec7"
server_key_version: "encryptionkey@0"
I20260808 13:01:42.273617 19326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "fde2a28d8b7840c89355dd65fa1c09de"
format_stamp: "Formatted at 2026-08-08 13:01:42 on dist-test-slave-tk9z"
server_key: "83abfa8d9ef3d5196b72cd7591f23a37"
server_key_iv: "0382cf36a87402a82cf94bf82f799ec7"
server_key_version: "encryptionkey@0"
I20260808 13:01:42.279055 19326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 13:01:42.282516 19347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:42.283799 19326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 13:01:42.283972 19326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "fde2a28d8b7840c89355dd65fa1c09de"
format_stamp: "Formatted at 2026-08-08 13:01:42 on dist-test-slave-tk9z"
server_key: "83abfa8d9ef3d5196b72cd7591f23a37"
server_key_iv: "0382cf36a87402a82cf94bf82f799ec7"
server_key_version: "encryptionkey@0"
I20260808 13:01:42.284175 19326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:42.298026 19326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:42.301893 19326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:42.302187 19326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:42.303128 19326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:42.304448 19326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:42.304551 19326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:42.304695 19326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:42.304771 19326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:42.338816 19326 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:36271
I20260808 13:01:42.338866 19529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:36271 every 8 connection(s)
I20260808 13:01:42.340140 19326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 13:01:42.346908 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 19326
I20260808 13:01:42.347144 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 13:01:42.347851 25244 external_mini_cluster.cc:1428] Setting key a981d0a7b4d9ff334158e75fbbd8101d
Aug 08 13:01:42 dist-test-slave-tk9z krb5kdc[19167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194102, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:01:42.362601 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:42.342767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:40825 (local address 127.24.167.62:42185)
0808 13:01:42.343243 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:42.343248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:42.343995 (+ 747us) server_negotiation.cc:407] Connection header received
0808 13:01:42.345845 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:42.345852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:42.345953 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:42.346077 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:42.347786 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:42.348700 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:42.350649 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:42.351072 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:42.351221 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:42.355684 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:42.355689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:42.355693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:42.355725 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:42.358492 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:42.359441 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:42.359447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:42.359450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:42.359570 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:42.360073 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:42.360076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:42.360078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:42.360470 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:42.360630 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:42.361702 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0808 13:01:42.362084 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":265,"threads_started":1}
I20260808 13:01:42.364432 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:42.343119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42185 (local address 127.24.167.1:40825)
0808 13:01:42.343822 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:42.343859 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:42.345559 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:42.346321 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:42.346330 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:42.346924 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:42.347607 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:42.347625 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:42.348894 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:42.348898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:42.350507 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:42.350521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:42.351328 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:42.351335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:42.351588 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:42.352393 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:01:42.352418 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:42.355435 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:42.358637 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:42.358648 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:42.358668 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:42.359319 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:42.359676 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:42.359683 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:42.359689 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:42.359958 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:42.360631 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:42.360642 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:42.361131 (+ 489us) client_negotiation.cc:769] Sending connection context
0808 13:01:42.361497 (+ 366us) client_negotiation.cc:240] Negotiation successful
0808 13:01:42.363381 (+ 1884us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":284,"threads_started":1}
I20260808 13:01:42.366173 19532 heartbeater.cc:347] Connected to a master server at 127.24.167.62:42185
I20260808 13:01:42.366531 19532 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:42.367282 19532 heartbeater.cc:511] Master 127.24.167.62:42185 requested a full tablet report, sending...
I20260808 13:01:42.369714 19235 ts_manager.cc:194] Registered new tserver with Master: fde2a28d8b7840c89355dd65fa1c09de (127.24.167.1:36271)
I20260808 13:01:42.372038 19235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:40825
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 13:01:42.418200 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42185
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:01:42.766185 19541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:42.766572 19541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:42.766746 19541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:42.774711 19541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:42.774870 19541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:42.775054 19541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 13:01:42.785168 19541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:42.786577 19541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:42.788024 19541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:42.801796 19552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:42.801796 19551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:42.801828 19554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:42.802910 19541 server_base.cc:1034] running on GCE node
I20260808 13:01:42.803650 19541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:42.804750 19541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:42.805986 19541 hybrid_clock.cc:648] HybridClock initialized: now 1786194102805924 us; error 61 us; skew 500 ppm
I20260808 13:01:42.806321 19541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:42 dist-test-slave-tk9z krb5kdc[19167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194102, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:42.809734 19541 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 13:01:42.811362 19541 webserver.cc:460] Webserver started at http://127.24.167.2:45437/ using document root <none> and password file <none>
I20260808 13:01:42.812119 19541 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:42.812275 19541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:42.812631 19541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:42.815757 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "788cdd6d51a4497db1cd4f774babacbf"
format_stamp: "Formatted at 2026-08-08 13:01:42 on dist-test-slave-tk9z"
server_key: "925628a25f498a0a3c9a209b395ec99a"
server_key_iv: "e02318194cd13f05bd1acef30c263e35"
server_key_version: "encryptionkey@0"
I20260808 13:01:42.816591 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "788cdd6d51a4497db1cd4f774babacbf"
format_stamp: "Formatted at 2026-08-08 13:01:42 on dist-test-slave-tk9z"
server_key: "925628a25f498a0a3c9a209b395ec99a"
server_key_iv: "e02318194cd13f05bd1acef30c263e35"
server_key_version: "encryptionkey@0"
I20260808 13:01:42.822114 19541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 13:01:42.826125 19562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:42.827780 19541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:01:42.828017 19541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "788cdd6d51a4497db1cd4f774babacbf"
format_stamp: "Formatted at 2026-08-08 13:01:42 on dist-test-slave-tk9z"
server_key: "925628a25f498a0a3c9a209b395ec99a"
server_key_iv: "e02318194cd13f05bd1acef30c263e35"
server_key_version: "encryptionkey@0"
I20260808 13:01:42.828264 19541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:42.860389 19541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:42.865684 19541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:42.866029 19541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:42.867234 19541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:42.868752 19541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:42.868886 19541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:42.869058 19541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:42.869140 19541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:42.905743 19541 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:33871
I20260808 13:01:42.905790 19730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:33871 every 8 connection(s)
I20260808 13:01:42.907291 19541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 13:01:42.910585 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 19541
I20260808 13:01:42.910811 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 13:01:42.911620 25244 external_mini_cluster.cc:1428] Setting key b87c02887563a02016b00ab11374e3b0
Aug 08 13:01:42 dist-test-slave-tk9z krb5kdc[19167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194102, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:01:42.936632 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:42.912965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:35721 (local address 127.24.167.62:42185)
0808 13:01:42.915005 (+ 2040us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:42.915012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:42.915871 (+ 859us) server_negotiation.cc:407] Connection header received
0808 13:01:42.918571 (+ 2700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:42.918577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:42.918678 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:42.918818 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:42.920475 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:42.921294 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:42.923200 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:42.923677 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:42.923829 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:42.928177 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:42.928183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:42.928187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:42.928249 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:42.932562 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:42.933608 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:42.933618 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:42.933626 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:42.933811 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:42.934267 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:42.934276 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:42.934282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:42.934815 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:42.935014 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:42.935737 (+ 723us) server_negotiation.cc:299] Negotiation successful
0808 13:01:42.936042 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1906,"thread_start_us":1774,"threads_started":1}
I20260808 13:01:42.937256 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:42.913606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42185 (local address 127.24.167.2:35721)
0808 13:01:42.915674 (+ 2068us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:42.915737 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:42.918209 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:42.919095 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:42.919104 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:42.919650 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:42.920283 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:42.920302 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:42.921479 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:42.921484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:42.923007 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:42.923020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:42.923955 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:42.923965 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:42.924215 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:42.925017 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:01:42.925042 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:42.927900 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:42.932705 (+ 4805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:42.932712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:42.932725 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:42.933448 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:42.933940 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:42.933943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:42.933945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:42.934132 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:42.934978 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:42.934984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:42.935308 (+ 324us) client_negotiation.cc:769] Sending connection context
0808 13:01:42.935562 (+ 254us) client_negotiation.cc:240] Negotiation successful
0808 13:01:42.935964 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1772,"thread_start_us":1519,"threads_started":1}
I20260808 13:01:42.938898 19731 heartbeater.cc:347] Connected to a master server at 127.24.167.62:42185
I20260808 13:01:42.939260 19731 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:42.940411 19731 heartbeater.cc:511] Master 127.24.167.62:42185 requested a full tablet report, sending...
I20260808 13:01:42.942237 19235 ts_manager.cc:194] Registered new tserver with Master: 788cdd6d51a4497db1cd4f774babacbf (127.24.167.2:33871)
I20260808 13:01:42.943150 19235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:35721
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 13:01:42.984983 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42185
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:01:43.333882 19745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:43.334187 19745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:43.334425 19745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:43.342734 19745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:43.342891 19745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:43.343067 19745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 13:01:43.354259 19745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:43.355806 19745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:43.357163 19745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:43.371434 19751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:43.371536 19753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:43.372032 19758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:43.372408 19745 server_base.cc:1034] running on GCE node
I20260808 13:01:43.373095 19745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:43.374079 19745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:43.375151 19532 heartbeater.cc:503] Master 127.24.167.62:42185 was elected leader, sending a full tablet report...
I20260808 13:01:43.375345 19745 hybrid_clock.cc:648] HybridClock initialized: now 1786194103375260 us; error 62 us; skew 500 ppm
I20260808 13:01:43.375801 19745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:43 dist-test-slave-tk9z krb5kdc[19167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194103, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:43.379477 19745 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 13:01:43.381011 19745 webserver.cc:460] Webserver started at http://127.24.167.3:37199/ using document root <none> and password file <none>
I20260808 13:01:43.381738 19745 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:43.381891 19745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:43.382221 19745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:43.385324 19745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "232f7696396e4183adf08d1e93d7d716"
format_stamp: "Formatted at 2026-08-08 13:01:43 on dist-test-slave-tk9z"
server_key: "5b3538c35c4c9717bc92b454c1307737"
server_key_iv: "7f96ce7cd50158dd55b1b910d83f1088"
server_key_version: "encryptionkey@0"
I20260808 13:01:43.386166 19745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "232f7696396e4183adf08d1e93d7d716"
format_stamp: "Formatted at 2026-08-08 13:01:43 on dist-test-slave-tk9z"
server_key: "5b3538c35c4c9717bc92b454c1307737"
server_key_iv: "7f96ce7cd50158dd55b1b910d83f1088"
server_key_version: "encryptionkey@0"
I20260808 13:01:43.391871 19745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 13:01:43.395586 19768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:43.397110 19745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 13:01:43.397292 19745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "232f7696396e4183adf08d1e93d7d716"
format_stamp: "Formatted at 2026-08-08 13:01:43 on dist-test-slave-tk9z"
server_key: "5b3538c35c4c9717bc92b454c1307737"
server_key_iv: "7f96ce7cd50158dd55b1b910d83f1088"
server_key_version: "encryptionkey@0"
I20260808 13:01:43.397508 19745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:43.413178 19745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:43.417327 19745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:43.417659 19745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:43.418745 19745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:43.420128 19745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:43.420235 19745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:43.420390 19745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:43.420466 19745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:43.454811 19745 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:46145
I20260808 13:01:43.454860 19956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:46145 every 8 connection(s)
I20260808 13:01:43.456303 19745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 13:01:43.465721 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 19745
I20260808 13:01:43.465903 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 13:01:43.466480 25244 external_mini_cluster.cc:1428] Setting key 711f12e97666bd3d96b89e7eeb1a5d1d
Aug 08 13:01:43 dist-test-slave-tk9z krb5kdc[19167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194103, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:01:43.477092 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.458879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:39849 (local address 127.24.167.62:42185)
0808 13:01:43.459342 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:43.459347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:43.460313 (+ 966us) server_negotiation.cc:407] Connection header received
0808 13:01:43.462103 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:43.462108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:43.462191 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:43.462294 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:43.464077 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.464872 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.466598 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.467195 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.467326 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:43.471587 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:43.471592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:43.471596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:43.471630 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:43.474234 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:43.474974 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:43.474978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:43.474980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:43.475106 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:43.475497 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:43.475500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:43.475502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:43.475773 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:43.475905 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:43.476617 (+ 712us) server_negotiation.cc:299] Negotiation successful
0808 13:01:43.476802 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":241,"threads_started":1}
I20260808 13:01:43.478092 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.459436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42185 (local address 127.24.167.3:39849)
0808 13:01:43.460151 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:43.460186 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:43.461840 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:43.462782 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:43.462790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:43.463302 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:43.463889 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:43.463905 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.465013 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.465017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:43.466442 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:43.466456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.467435 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.467442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:43.467729 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:43.468578 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:01:43.468609 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:43.471381 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:43.474362 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:43.474410 (+ 48us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:43.474434 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:43.474855 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:43.475201 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:43.475204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:43.475206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:43.475389 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:43.475879 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:43.475884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:43.476228 (+ 344us) client_negotiation.cc:769] Sending connection context
0808 13:01:43.476486 (+ 258us) client_negotiation.cc:240] Negotiation successful
0808 13:01:43.476928 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":269,"threads_started":1}
I20260808 13:01:43.479588 19957 heartbeater.cc:347] Connected to a master server at 127.24.167.62:42185
I20260808 13:01:43.479904 19957 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:43.480787 19957 heartbeater.cc:511] Master 127.24.167.62:42185 requested a full tablet report, sending...
I20260808 13:01:43.482339 19235 ts_manager.cc:194] Registered new tserver with Master: 232f7696396e4183adf08d1e93d7d716 (127.24.167.3:46145)
I20260808 13:01:43.482969 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 13:01:43.483345 19235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:39849
I20260808 13:01:43.499953 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.487853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.24.167.62:42185)
0808 13:01:43.488089 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:43.488097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:43.488412 (+ 315us) server_negotiation.cc:407] Connection header received
0808 13:01:43.488576 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:43.488580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:43.488650 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:43.488730 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:43.490316 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.491089 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.492348 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.492736 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.492822 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:43.495134 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:43.495138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:43.495141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:43.495171 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:43.497538 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:43.498124 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:43.498128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:43.498130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:43.498247 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:43.498639 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:43.498643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:43.498645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:43.498973 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:43.499154 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:43.499435 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 13:01:43.499606 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 13:01:43.503986 19235 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 13:01:43.510926 19235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 13:01:43.534044 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.523223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.24.167.1:36271)
0808 13:01:43.524381 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:43.524388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:43.524435 (+ 47us) server_negotiation.cc:407] Connection header received
0808 13:01:43.524542 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:43.524550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:43.524792 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:43.525025 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:43.526816 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.528460 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.531563 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.532748 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.532900 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:43.533139 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:43.533536 (+ 397us) server_negotiation.cc:299] Negotiation successful
0808 13:01:43.533708 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1001,"thread_start_us":845,"threads_started":1}
I20260808 13:01:43.534449 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.524365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:33871 (local address 127.0.0.1:41444)
0808 13:01:43.524926 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:43.524958 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:43.525070 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:43.526874 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:43.526881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:43.526910 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:43.527353 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:43.527366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.529231 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.529238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:43.531697 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:43.531711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.533568 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.533574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:43.533735 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:43.533763 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 13:01:43.533824 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 13:01:43.533952 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":77,"thread_start_us":303,"threads_started":1}
I20260808 13:01:43.534183 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.523065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:36271 (local address 127.0.0.1:55064)
0808 13:01:43.523852 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:43.523902 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:43.524109 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:43.525424 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:43.525436 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:43.525482 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:43.526585 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:43.526600 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.528607 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.528613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:43.531370 (+ 2757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:43.531385 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.533031 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.533039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:43.533252 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:43.533337 (+ 85us) client_negotiation.cc:769] Sending connection context
0808 13:01:43.533480 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 13:01:43.533681 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":75,"thread_start_us":401,"threads_started":1}
I20260808 13:01:43.535099 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.524635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.24.167.2:33871)
0808 13:01:43.526270 (+ 1635us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:43.526281 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:43.526309 (+ 28us) server_negotiation.cc:407] Connection header received
0808 13:01:43.526436 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:43.526442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:43.526670 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:43.527617 (+ 947us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:43.527850 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.529113 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.531855 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.533318 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.534275 (+ 957us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:43.534515 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:43.534584 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 13:01:43.534761 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1437,"thread_start_us":1319,"threads_started":1}
I20260808 13:01:43.535336 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.525004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:46145 (local address 127.0.0.1:57116)
0808 13:01:43.525592 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:43.525615 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:43.525738 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:43.527184 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:43.527188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:43.527216 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:43.527664 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:43.527676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.529192 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.529201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:43.531246 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:43.531259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.533283 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.533289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:43.533511 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:43.533560 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 13:01:43.533642 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 13:01:43.534969 (+ 1327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":9,"thread_start_us":891,"threads_started":1}
I20260808 13:01:43.535938 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.525607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.24.167.3:46145)
0808 13:01:43.526631 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:43.526639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:43.526671 (+ 32us) server_negotiation.cc:407] Connection header received
0808 13:01:43.526770 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:43.526776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:43.527002 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:43.527198 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:43.527827 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.529026 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.531433 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.532754 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.532883 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:43.533129 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:43.535575 (+ 2446us) server_negotiation.cc:299] Negotiation successful
0808 13:01:43.535706 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":669,"threads_started":1}
I20260808 13:01:43.539047 19638 tablet_service.cc:1467] Processing CreateTablet for tablet d25c228c4eb24b8585b0fd221249fd46 (DEFAULT_TABLE table=test-table [id=f3fa4fb2a25e429e8d6b54805fae5772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:43.539042 19420 tablet_service.cc:1467] Processing CreateTablet for tablet d25c228c4eb24b8585b0fd221249fd46 (DEFAULT_TABLE table=test-table [id=f3fa4fb2a25e429e8d6b54805fae5772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:43.539249 19842 tablet_service.cc:1467] Processing CreateTablet for tablet d25c228c4eb24b8585b0fd221249fd46 (DEFAULT_TABLE table=test-table [id=f3fa4fb2a25e429e8d6b54805fae5772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:43.540454 19638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25c228c4eb24b8585b0fd221249fd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:43.540586 19842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25c228c4eb24b8585b0fd221249fd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:43.540586 19420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25c228c4eb24b8585b0fd221249fd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:43.554281 19987 tablet_bootstrap.cc:492] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716: Bootstrap starting.
I20260808 13:01:43.557848 19987 tablet_bootstrap.cc:654] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:43.559168 19987 log.cc:824] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:43.562003 19987 tablet_bootstrap.cc:492] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716: No bootstrap required, opened a new log
I20260808 13:01:43.562460 19987 ts_tablet_manager.cc:1397] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20260808 13:01:43.562484 19990 tablet_bootstrap.cc:492] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf: Bootstrap starting.
I20260808 13:01:43.564265 19991 tablet_bootstrap.cc:492] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de: Bootstrap starting.
I20260808 13:01:43.567178 19990 tablet_bootstrap.cc:654] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:43.567703 19991 tablet_bootstrap.cc:654] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:43.568526 19990 log.cc:824] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:43.568768 19991 log.cc:824] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:43.572008 19991 tablet_bootstrap.cc:492] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de: No bootstrap required, opened a new log
I20260808 13:01:43.572016 19990 tablet_bootstrap.cc:492] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf: No bootstrap required, opened a new log
I20260808 13:01:43.572304 19991 ts_tablet_manager.cc:1397] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 13:01:43.572304 19990 ts_tablet_manager.cc:1397] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260808 13:01:43.572990 19987 raft_consensus.cc:348] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde2a28d8b7840c89355dd65fa1c09de" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36271 } } peers { permanent_uuid: "788cdd6d51a4497db1cd4f774babacbf" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33871 } } peers { permanent_uuid: "232f7696396e4183adf08d1e93d7d716" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 46145 } }
I20260808 13:01:43.573374 19987 raft_consensus.cc:374] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:43.573483 19987 raft_consensus.cc:729] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 232f7696396e4183adf08d1e93d7d716, State: Initialized, Role: FOLLOWER
I20260808 13:01:43.574116 19987 consensus_queue.cc:260] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde2a28d8b7840c89355dd65fa1c09de" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36271 } } peers { permanent_uuid: "788cdd6d51a4497db1cd4f774babacbf" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33871 } } peers { permanent_uuid: "232f7696396e4183adf08d1e93d7d716" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 46145 } }
I20260808 13:01:43.575229 19957 heartbeater.cc:503] Master 127.24.167.62:42185 was elected leader, sending a full tablet report...
I20260808 13:01:43.575893 19987 ts_tablet_manager.cc:1428] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260808 13:01:43.583352 19991 raft_consensus.cc:348] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde2a28d8b7840c89355dd65fa1c09de" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36271 } } peers { permanent_uuid: "788cdd6d51a4497db1cd4f774babacbf" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33871 } } peers { permanent_uuid: "232f7696396e4183adf08d1e93d7d716" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 46145 } }
I20260808 13:01:43.583352 19990 raft_consensus.cc:348] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde2a28d8b7840c89355dd65fa1c09de" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36271 } } peers { permanent_uuid: "788cdd6d51a4497db1cd4f774babacbf" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33871 } } peers { permanent_uuid: "232f7696396e4183adf08d1e93d7d716" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 46145 } }
I20260808 13:01:43.583863 19990 raft_consensus.cc:374] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:43.583863 19991 raft_consensus.cc:374] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:43.583981 19990 raft_consensus.cc:729] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788cdd6d51a4497db1cd4f774babacbf, State: Initialized, Role: FOLLOWER
I20260808 13:01:43.583976 19991 raft_consensus.cc:729] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde2a28d8b7840c89355dd65fa1c09de, State: Initialized, Role: FOLLOWER
I20260808 13:01:43.584633 19991 consensus_queue.cc:260] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde2a28d8b7840c89355dd65fa1c09de" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36271 } } peers { permanent_uuid: "788cdd6d51a4497db1cd4f774babacbf" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33871 } } peers { permanent_uuid: "232f7696396e4183adf08d1e93d7d716" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 46145 } }
I20260808 13:01:43.584627 19990 consensus_queue.cc:260] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde2a28d8b7840c89355dd65fa1c09de" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36271 } } peers { permanent_uuid: "788cdd6d51a4497db1cd4f774babacbf" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33871 } } peers { permanent_uuid: "232f7696396e4183adf08d1e93d7d716" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 46145 } }
I20260808 13:01:43.586473 19731 heartbeater.cc:503] Master 127.24.167.62:42185 was elected leader, sending a full tablet report...
I20260808 13:01:43.586732 19991 ts_tablet_manager.cc:1428] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260808 13:01:43.587050 19990 ts_tablet_manager.cc:1428] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260808 13:01:43.592512 19534 tablet.cc:2369] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 13:01:43.664753 19732 tablet.cc:2369] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:01:43.706311 19997 raft_consensus.cc:482] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 13:01:43.706717 19997 raft_consensus.cc:504] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde2a28d8b7840c89355dd65fa1c09de" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36271 } } peers { permanent_uuid: "788cdd6d51a4497db1cd4f774babacbf" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33871 } } peers { permanent_uuid: "232f7696396e4183adf08d1e93d7d716" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 46145 } }
W20260808 13:01:43.708016 19958 tablet.cc:2369] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:01:43.708600 19997 leader_election.cc:288] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde2a28d8b7840c89355dd65fa1c09de (127.24.167.1:36271), 232f7696396e4183adf08d1e93d7d716 (127.24.167.3:46145)
I20260808 13:01:43.716673 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.710138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:53071 (local address 127.24.167.3:46145)
0808 13:01:43.710700 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:43.710706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:43.710760 (+ 54us) server_negotiation.cc:407] Connection header received
0808 13:01:43.710994 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:43.711001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:43.711078 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:43.711189 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:43.711787 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.712777 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.714600 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.715684 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.715858 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:43.715937 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:43.716276 (+ 339us) server_negotiation.cc:299] Negotiation successful
0808 13:01:43.716404 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417}
I20260808 13:01:43.716723 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.709895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:46145 (local address 127.24.167.2:53071)
0808 13:01:43.710658 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:43.710676 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:43.710831 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:43.711222 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:43.711230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:43.711255 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:43.711616 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:43.711627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.712937 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.712941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:43.714449 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:43.714461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.715912 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.715919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:43.716096 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:43.716131 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 13:01:43.716224 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 13:01:43.716350 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":460,"threads_started":1}
I20260808 13:01:43.717299 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.709127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:47405 (local address 127.24.167.1:36271)
0808 13:01:43.709362 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:43.709368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:43.709844 (+ 476us) server_negotiation.cc:407] Connection header received
0808 13:01:43.710224 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:43.710232 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:43.710360 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:43.710450 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:43.711400 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.712435 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.714992 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.716150 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.716329 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:43.716394 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:43.716969 (+ 575us) server_negotiation.cc:299] Negotiation successful
0808 13:01:43.717099 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 13:01:43.718178 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.708974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:36271 (local address 127.24.167.2:47405)
0808 13:01:43.709742 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:43.709763 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:43.709993 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:43.710697 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:43.710701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:43.710733 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:43.711256 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:43.711270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.712582 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.712587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:43.714822 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:43.714836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.716563 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.716569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:43.716761 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:43.716811 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 13:01:43.717639 (+ 828us) client_negotiation.cc:240] Negotiation successful
0808 13:01:43.717779 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"spinlock_wait_cycles":16768,"thread_start_us":421,"threads_started":1}
I20260808 13:01:43.718030 19876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25c228c4eb24b8585b0fd221249fd46" candidate_uuid: "788cdd6d51a4497db1cd4f774babacbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "232f7696396e4183adf08d1e93d7d716" is_pre_election: true
I20260808 13:01:43.718703 19876 raft_consensus.cc:2454] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 788cdd6d51a4497db1cd4f774babacbf in term 0.
I20260808 13:01:43.719441 19565 leader_election.cc:302] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 232f7696396e4183adf08d1e93d7d716, 788cdd6d51a4497db1cd4f774babacbf; no voters:
I20260808 13:01:43.719342 19451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25c228c4eb24b8585b0fd221249fd46" candidate_uuid: "788cdd6d51a4497db1cd4f774babacbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde2a28d8b7840c89355dd65fa1c09de" is_pre_election: true
I20260808 13:01:43.719856 19997 raft_consensus.cc:2781] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 13:01:43.719884 19451 raft_consensus.cc:2454] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 788cdd6d51a4497db1cd4f774babacbf in term 0.
I20260808 13:01:43.720027 19997 raft_consensus.cc:482] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 13:01:43.720170 19997 raft_consensus.cc:3037] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:43.722642 19997 raft_consensus.cc:504] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde2a28d8b7840c89355dd65fa1c09de" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36271 } } peers { permanent_uuid: "788cdd6d51a4497db1cd4f774babacbf" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33871 } } peers { permanent_uuid: "232f7696396e4183adf08d1e93d7d716" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 46145 } }
I20260808 13:01:43.723248 19997 leader_election.cc:288] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [CANDIDATE]: Term 1 election: Requested vote from peers fde2a28d8b7840c89355dd65fa1c09de (127.24.167.1:36271), 232f7696396e4183adf08d1e93d7d716 (127.24.167.3:46145)
I20260808 13:01:43.723759 19451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25c228c4eb24b8585b0fd221249fd46" candidate_uuid: "788cdd6d51a4497db1cd4f774babacbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde2a28d8b7840c89355dd65fa1c09de"
I20260808 13:01:43.723917 19876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25c228c4eb24b8585b0fd221249fd46" candidate_uuid: "788cdd6d51a4497db1cd4f774babacbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "232f7696396e4183adf08d1e93d7d716"
I20260808 13:01:43.724011 19451 raft_consensus.cc:3037] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:43.724166 19876 raft_consensus.cc:3037] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:43.726349 19451 raft_consensus.cc:2454] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 788cdd6d51a4497db1cd4f774babacbf in term 1.
I20260808 13:01:43.726452 19876 raft_consensus.cc:2454] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 788cdd6d51a4497db1cd4f774babacbf in term 1.
I20260808 13:01:43.726866 19565 leader_election.cc:302] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 788cdd6d51a4497db1cd4f774babacbf, fde2a28d8b7840c89355dd65fa1c09de; no voters:
I20260808 13:01:43.727207 19997 raft_consensus.cc:2781] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:01:43.727746 19997 raft_consensus.cc:686] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [term 1 LEADER]: Becoming Leader. State: Replica: 788cdd6d51a4497db1cd4f774babacbf, State: Running, Role: LEADER
I20260808 13:01:43.728271 19997 consensus_queue.cc:237] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde2a28d8b7840c89355dd65fa1c09de" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36271 } } peers { permanent_uuid: "788cdd6d51a4497db1cd4f774babacbf" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33871 } } peers { permanent_uuid: "232f7696396e4183adf08d1e93d7d716" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 46145 } }
I20260808 13:01:43.733848 19234 catalog_manager.cc:5495] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 788cdd6d51a4497db1cd4f774babacbf (127.24.167.2). New cstate: current_term: 1 leader_uuid: "788cdd6d51a4497db1cd4f774babacbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde2a28d8b7840c89355dd65fa1c09de" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "788cdd6d51a4497db1cd4f774babacbf" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 33871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "232f7696396e4183adf08d1e93d7d716" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 46145 } health_report { overall_health: UNKNOWN } } }
I20260808 13:01:43.777967 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.772874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.24.167.2:33871)
0808 13:01:43.773068 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:43.773072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:43.773104 (+ 32us) server_negotiation.cc:407] Connection header received
0808 13:01:43.773197 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:43.773200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:43.773280 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:43.773404 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:01:43.773763 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.774559 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.775972 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.776367 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.776457 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:43.776929 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:01:43.777446 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:01:43.777564 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:43.777657 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 13:01:43.777727 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 13:01:43.786288 19876 raft_consensus.cc:1261] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716 [term 1 FOLLOWER]: Refusing update from remote peer 788cdd6d51a4497db1cd4f774babacbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:01:43.786443 19451 raft_consensus.cc:1261] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de [term 1 FOLLOWER]: Refusing update from remote peer 788cdd6d51a4497db1cd4f774babacbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:01:43.787477 20000 consensus_queue.cc:1035] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "232f7696396e4183adf08d1e93d7d716" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 46145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:01:43.787845 19997 consensus_queue.cc:1035] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde2a28d8b7840c89355dd65fa1c09de" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:01:43.803500 20006 mvcc.cc:205] Tried to move back new op lower bound from 7316251049098555392 to 7316251048875102208. Current Snapshot: MvccSnapshot[applied={T|T < 7316251049098555392}]
I20260808 13:01:43.814927 19234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:01:43.815237 19234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:01:43.819161 19234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7b86038097a41928d4fd3ad4515ea88: Sending DeleteTablet for 3 replicas of tablet d25c228c4eb24b8585b0fd221249fd46
I20260808 13:01:43.820442 19420 tablet_service.cc:1514] Processing DeleteTablet for tablet d25c228c4eb24b8585b0fd221249fd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:43 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:55064
I20260808 13:01:43.820645 19842 tablet_service.cc:1514] Processing DeleteTablet for tablet d25c228c4eb24b8585b0fd221249fd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:43 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:57116
I20260808 13:01:43.820645 19638 tablet_service.cc:1514] Processing DeleteTablet for tablet d25c228c4eb24b8585b0fd221249fd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:43 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:41444
I20260808 13:01:43.823112 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 19326
I20260808 13:01:43.840286 20021 tablet_replica.cc:330] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf: stopping tablet replica
I20260808 13:01:43.841101 20021 raft_consensus.cc:2229] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [term 1 LEADER]: Raft consensus shutting down.
I20260808 13:01:43.842037 20021 raft_consensus.cc:2258] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:43.843488 20020 tablet_replica.cc:330] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716: stopping tablet replica
I20260808 13:01:43.843986 20020 raft_consensus.cc:2229] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:01:43.844537 20020 raft_consensus.cc:2258] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:43.846441 20021 ts_tablet_manager.cc:1905] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:43.847393 20020 ts_tablet_manager.cc:1905] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 13:01:43 dist-test-slave-tk9z krb5kdc[19167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 13:01:43.849481 20019 tablet_replica.cc:330] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de: stopping tablet replica
I20260808 13:01:43.850106 20019 raft_consensus.cc:2229] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:01:43.850811 20019 raft_consensus.cc:2258] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:43.855603 20020 ts_tablet_manager.cc:1918] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:43.855862 20020 log.cc:1196] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/d25c228c4eb24b8585b0fd221249fd46
I20260808 13:01:43.855898 20019 ts_tablet_manager.cc:1905] T d25c228c4eb24b8585b0fd221249fd46 P fde2a28d8b7840c89355dd65fa1c09de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:43.856582 20020 ts_tablet_manager.cc:1939] T d25c228c4eb24b8585b0fd221249fd46 P 232f7696396e4183adf08d1e93d7d716: Deleting consensus metadata
I20260808 13:01:43.858721 19212 catalog_manager.cc:4841] TS 232f7696396e4183adf08d1e93d7d716 (127.24.167.3:46145): tablet d25c228c4eb24b8585b0fd221249fd46 (table test-table [id=f3fa4fb2a25e429e8d6b54805fae5772]) successfully deleted
I20260808 13:01:43.859223 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.824185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.24.167.1:36271)
0808 13:01:43.824426 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:43.824437 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:43.831220 (+ 6783us) server_negotiation.cc:407] Connection header received
0808 13:01:43.832530 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:43.832540 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:43.832661 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:43.832774 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:43.837632 (+ 4858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.839427 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.843888 (+ 4461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.844430 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:43.844592 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:43.850489 (+ 5897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:43.850502 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:43.850520 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:43.850572 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:43.855101 (+ 4529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:43.856251 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:43.856261 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:43.856266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:43.856468 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:43.857037 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:43.857042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:43.857045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:43.857551 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:43.857875 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:43.858533 (+ 658us) server_negotiation.cc:299] Negotiation successful
0808 13:01:43.858809 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":75}
I20260808 13:01:43.860625 19519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:43.863095 20021 ts_tablet_manager.cc:1918] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:43.863459 20021 log.cc:1196] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/d25c228c4eb24b8585b0fd221249fd46
I20260808 13:01:43.864374 20021 ts_tablet_manager.cc:1939] T d25c228c4eb24b8585b0fd221249fd46 P 788cdd6d51a4497db1cd4f774babacbf: Deleting consensus metadata
I20260808 13:01:43.866454 19212 catalog_manager.cc:4841] TS 788cdd6d51a4497db1cd4f774babacbf (127.24.167.2:33871): tablet d25c228c4eb24b8585b0fd221249fd46 (table test-table [id=f3fa4fb2a25e429e8d6b54805fae5772]) successfully deleted
I20260808 13:01:43.968693 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 19326
I20260808 13:01:43.993857 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 19541
W20260808 13:01:43.993952 19212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.167.1:36271 (error 108)
I20260808 13:01:43.996109 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.995618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:36271 (local address 127.0.0.1:55080)
0808 13:01:43.995811 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:43.995955 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260808 13:01:43.996451 19212 catalog_manager.cc:4584] TS fde2a28d8b7840c89355dd65fa1c09de (127.24.167.1:36271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d25c228c4eb24b8585b0fd221249fd46: Network error: Client connection negotiation failed: client connection to 127.24.167.1:36271: connect: Connection refused (error 111)
Aug 08 13:01:43 dist-test-slave-tk9z krb5kdc[19167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 13:01:44.013734 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:43.995297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.24.167.2:33871)
0808 13:01:43.995473 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:43.995477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:43.995495 (+ 18us) server_negotiation.cc:407] Connection header received
0808 13:01:43.995555 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:43.995558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:43.995642 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:43.995773 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:43.997552 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:43.998732 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:44.000642 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:44.001179 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:44.001287 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:44.006753 (+ 5466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:44.006768 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:44.006788 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:44.006832 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:44.010578 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:44.011378 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:44.011388 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:44.011392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:44.011555 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:44.011982 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:44.011987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:44.011991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:44.012388 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:44.012744 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:44.013069 (+ 325us) server_negotiation.cc:299] Negotiation successful
0808 13:01:44.013346 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 13:01:44.014551 19723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:44.050066 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:44.049609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:36271 (local address 127.0.0.1:55088)
0808 13:01:44.049805 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:44.049909 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 13:01:44.051188 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:44.050818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:36271 (local address 127.0.0.1:55096)
0808 13:01:44.050980 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:44.051073 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 13:01:44.112239 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:44.111682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:36271 (local address 127.0.0.1:55110)
0808 13:01:44.111902 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:44.112008 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 13:01:44.113328 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:44.112965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:36271 (local address 127.0.0.1:55112)
0808 13:01:44.113124 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:44.113202 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 13:01:44.147346 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 19541
I20260808 13:01:44.169054 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 19745
Aug 08 13:01:44 dist-test-slave-tk9z krb5kdc[19167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 13:01:44.185104 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:44.169901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.24.167.3:46145)
0808 13:01:44.170093 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:44.170101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:44.170123 (+ 22us) server_negotiation.cc:407] Connection header received
0808 13:01:44.170396 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:44.170404 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:44.170513 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:44.170634 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:44.172199 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:44.172927 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:44.174342 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:44.174881 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:44.175029 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:44.179619 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:44.179636 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:44.179653 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:44.179688 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:44.182644 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:44.183272 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:44.183280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:44.183282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:44.183439 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:44.183781 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:44.183784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:44.183786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:44.184125 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:44.184336 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:44.184682 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 13:01:44.184843 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 13:01:44.185765 19950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:44.194248 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:44.193776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:36271 (local address 127.0.0.1:55128)
0808 13:01:44.193981 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:44.194073 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 13:01:44.195348 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:44.194966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:36271 (local address 127.0.0.1:55134)
0808 13:01:44.195162 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:44.195238 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260808 13:01:44.276899 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 19745
I20260808 13:01:44.300740 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 19183
I20260808 13:01:44.302052 19276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:44.416394 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 19183
2026-08-08T13:01:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4833 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T13:01:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:01:44Z Disabled control of system clock
I20260808 13:01:44.521230 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:44325
--webserver_interface=127.24.167.62
--webserver_port=0
--builtin_ntp_servers=127.24.167.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 13:01:44.879703 20034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:44.880004 20034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:44.880091 20034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:44.887481 20034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:01:44.887604 20034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:44.887643 20034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:44.887703 20034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:01:44.887748 20034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:01:44.897367 20034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:44325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:44325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:44.898700 20034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:44.900590 20034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:44.915282 20044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:44.915248 20050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:44.915215 20045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:44.916236 20034 server_base.cc:1034] running on GCE node
I20260808 13:01:44.917091 20034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:44.918640 20034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:44.919890 20034 hybrid_clock.cc:648] HybridClock initialized: now 1786194104919809 us; error 58 us; skew 500 ppm
I20260808 13:01:44.920349 20034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 13:01:44.926290 20034 webserver.cc:460] Webserver started at http://127.24.167.62:42433/ using document root <none> and password file <none>
I20260808 13:01:44.927136 20034 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:44.927237 20034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:44.927538 20034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:44.930819 20034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "746589a9de0f4d1cb3938dee512ac557"
format_stamp: "Formatted at 2026-08-08 13:01:44 on dist-test-slave-tk9z"
server_key: "5d88dc29d975d8c4b295176a6596e3ff"
server_key_iv: "c96b4c262a33e2e7c17f32abd33ea6f5"
server_key_version: "encryptionkey@0"
I20260808 13:01:44.931756 20034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "746589a9de0f4d1cb3938dee512ac557"
format_stamp: "Formatted at 2026-08-08 13:01:44 on dist-test-slave-tk9z"
server_key: "5d88dc29d975d8c4b295176a6596e3ff"
server_key_iv: "c96b4c262a33e2e7c17f32abd33ea6f5"
server_key_version: "encryptionkey@0"
I20260808 13:01:44.937345 20034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 13:01:44.941257 20060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:44.942848 20034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:01:44.943055 20034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "746589a9de0f4d1cb3938dee512ac557"
format_stamp: "Formatted at 2026-08-08 13:01:44 on dist-test-slave-tk9z"
server_key: "5d88dc29d975d8c4b295176a6596e3ff"
server_key_iv: "c96b4c262a33e2e7c17f32abd33ea6f5"
server_key_version: "encryptionkey@0"
I20260808 13:01:44.943305 20034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:44.955917 20034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:44.956825 20034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:44.957157 20034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:44.977362 20034 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:44325
I20260808 13:01:44.977385 20139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:44325 every 8 connection(s)
I20260808 13:01:44.978847 20034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:01:44.982575 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 20034
I20260808 13:01:44.982775 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:01:44.983042 20143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:44.983309 25244 external_mini_cluster.cc:1428] Setting key 77a2f603f35ff2ee98bf3d404fbcc9d5
I20260808 13:01:44.997876 20143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557: Bootstrap starting.
I20260808 13:01:44.998206 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:44.984990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.24.167.62:44325)
0808 13:01:44.986002 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:44.986012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:44.986053 (+ 41us) server_negotiation.cc:407] Connection header received
0808 13:01:44.988329 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:44.988356 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:44.988827 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:44.989215 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:44.990010 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:44.991488 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:44.993436 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:44.994152 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:44.994293 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:44.995682 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:44.995690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:44.995707 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:44.996282 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:44.996363 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:44.996385 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:44.996485 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:44.996741 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 13:01:44.997147 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":403,"threads_started":1}
I20260808 13:01:45.001806 20143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:45.002972 20143 log.cc:824] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:45.006278 20143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557: No bootstrap required, opened a new log
I20260808 13:01:45.014313 20143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "746589a9de0f4d1cb3938dee512ac557" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44325 } }
I20260808 13:01:45.014796 20143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:45.014876 20143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 746589a9de0f4d1cb3938dee512ac557, State: Initialized, Role: FOLLOWER
I20260808 13:01:45.015396 20143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "746589a9de0f4d1cb3938dee512ac557" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44325 } }
I20260808 13:01:45.015620 20143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:01:45.015719 20143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:01:45.015848 20143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:45.017990 20143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "746589a9de0f4d1cb3938dee512ac557" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44325 } }
I20260808 13:01:45.018549 20143 leader_election.cc:302] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 746589a9de0f4d1cb3938dee512ac557; no voters:
I20260808 13:01:45.019150 20143 leader_election.cc:288] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:01:45.019307 20150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:01:45.020561 20150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [term 1 LEADER]: Becoming Leader. State: Replica: 746589a9de0f4d1cb3938dee512ac557, State: Running, Role: LEADER
I20260808 13:01:45.021165 20150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "746589a9de0f4d1cb3938dee512ac557" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44325 } }
I20260808 13:01:45.021449 20143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:01:45.024387 20152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 746589a9de0f4d1cb3938dee512ac557. Latest consensus state: current_term: 1 leader_uuid: "746589a9de0f4d1cb3938dee512ac557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "746589a9de0f4d1cb3938dee512ac557" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44325 } } }
I20260808 13:01:45.024492 20151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "746589a9de0f4d1cb3938dee512ac557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "746589a9de0f4d1cb3938dee512ac557" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44325 } } }
I20260808 13:01:45.024849 20152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [sys.catalog]: This master's current role is: LEADER
I20260808 13:01:45.024897 20151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557 [sys.catalog]: This master's current role is: LEADER
I20260808 13:01:45.027035 20165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:01:45.032555 20165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:01:45.041929 20165 catalog_manager.cc:1347] Generated new cluster ID: e630a15f173f4eaca9368bcc9e076ca3
I20260808 13:01:45.042076 20165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:01:45.059388 20165 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:01:45.060767 20165 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:01:45.067731 20165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557: Generated new TSK 0
I20260808 13:01:45.068879 20165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:01:45.084719 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44325
--builtin_ntp_servers=127.24.167.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 13:01:45.440382 20180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:45.440675 20180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:45.440842 20180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:45.448442 20180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:45.448542 20180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:45.448738 20180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 13:01:45.457806 20180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:45.459133 20180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:45.460997 20180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:45.474658 20189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:45.474735 20194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:45.474967 20190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:45.475580 20180 server_base.cc:1034] running on GCE node
I20260808 13:01:45.476131 20180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:45.477094 20180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:45.478303 20180 hybrid_clock.cc:648] HybridClock initialized: now 1786194105478257 us; error 41 us; skew 500 ppm
I20260808 13:01:45.478685 20180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 13:01:45.480870 20180 webserver.cc:460] Webserver started at http://127.24.167.1:42723/ using document root <none> and password file <none>
I20260808 13:01:45.481559 20180 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:45.481675 20180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:45.482012 20180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:45.484915 20180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "7207bc99078144958504f9069c6261cf"
format_stamp: "Formatted at 2026-08-08 13:01:45 on dist-test-slave-tk9z"
server_key: "3b1c15ab4aa69f86abfe8763aa81a0e4"
server_key_iv: "90ba849c523ae0b64db4381d7edbf682"
server_key_version: "encryptionkey@0"
I20260808 13:01:45.485785 20180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "7207bc99078144958504f9069c6261cf"
format_stamp: "Formatted at 2026-08-08 13:01:45 on dist-test-slave-tk9z"
server_key: "3b1c15ab4aa69f86abfe8763aa81a0e4"
server_key_iv: "90ba849c523ae0b64db4381d7edbf682"
server_key_version: "encryptionkey@0"
I20260808 13:01:45.490926 20180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260808 13:01:45.494205 20202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:45.495501 20180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 13:01:45.495682 20180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "7207bc99078144958504f9069c6261cf"
format_stamp: "Formatted at 2026-08-08 13:01:45 on dist-test-slave-tk9z"
server_key: "3b1c15ab4aa69f86abfe8763aa81a0e4"
server_key_iv: "90ba849c523ae0b64db4381d7edbf682"
server_key_version: "encryptionkey@0"
I20260808 13:01:45.495901 20180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:45.512100 20180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:45.512938 20180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:45.513265 20180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:45.514261 20180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:45.515848 20180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:45.515990 20180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:45.516121 20180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:45.516211 20180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:45.549975 20180 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:45485
I20260808 13:01:45.550033 20386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:45485 every 8 connection(s)
I20260808 13:01:45.551477 20180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 13:01:45.557602 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 20180
I20260808 13:01:45.557821 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 13:01:45.558256 25244 external_mini_cluster.cc:1428] Setting key 11363f81608cb5ac81d4ad4980ab8ace
I20260808 13:01:45.560940 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44325
--builtin_ntp_servers=127.24.167.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 13:01:45.563989 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:45.554063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:40051 (local address 127.24.167.62:44325)
0808 13:01:45.554568 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:45.554573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:45.555604 (+ 1031us) server_negotiation.cc:407] Connection header received
0808 13:01:45.557410 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:45.557415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:45.557519 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:45.557642 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:45.558836 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:45.559597 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:45.561124 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:45.561552 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:45.561689 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:45.563126 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:45.563134 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:45.563143 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:45.563159 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:45.563201 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:45.563212 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:45.563310 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:45.563543 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 13:01:45.563642 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":264,"threads_started":1}
I20260808 13:01:45.564931 20393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:45.554578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:44325 (local address 127.24.167.1:40051)
0808 13:01:45.555445 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:45.555489 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:45.557145 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:45.557876 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:45.557884 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:45.558019 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:45.558661 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:45.558680 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:45.559757 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:45.559762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:45.560978 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:45.560990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:45.561794 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:45.561802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:45.562035 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:45.562777 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 13:01:45.562780 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:45.562818 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 13:01:45.562920 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:45.563310 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:45.563325 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:45.563335 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 13:01:45.563512 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 13:01:45.563885 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":271,"threads_started":1}
I20260808 13:01:45.566529 20389 heartbeater.cc:347] Connected to a master server at 127.24.167.62:44325
I20260808 13:01:45.566865 20389 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:45.567706 20389 heartbeater.cc:511] Master 127.24.167.62:44325 requested a full tablet report, sending...
I20260808 13:01:45.569765 20086 ts_manager.cc:194] Registered new tserver with Master: 7207bc99078144958504f9069c6261cf (127.24.167.1:45485)
I20260808 13:01:45.571369 20086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.167.1:40051
W20260808 13:01:45.926611 20394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:45.926949 20394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:45.927045 20394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:45.934676 20394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:45.934779 20394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:45.934970 20394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 13:01:45.944204 20394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:45.945550 20394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:45.947505 20394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:45.960654 20401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:45.960760 20403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:45.960624 20399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:45.961411 20394 server_base.cc:1034] running on GCE node
I20260808 13:01:45.962142 20394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:45.963174 20394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:45.964383 20394 hybrid_clock.cc:648] HybridClock initialized: now 1786194105964334 us; error 36 us; skew 500 ppm
I20260808 13:01:45.964720 20394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 13:01:45.967038 20394 webserver.cc:460] Webserver started at http://127.24.167.2:40697/ using document root <none> and password file <none>
I20260808 13:01:45.967727 20394 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:45.967857 20394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:45.968195 20394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:45.971190 20394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "c436192bb059470194f6d744dd84c0e1"
format_stamp: "Formatted at 2026-08-08 13:01:45 on dist-test-slave-tk9z"
server_key: "c1f81a0ef677d1105fe10b24ab316531"
server_key_iv: "658398722354004b36fc50bfb068c98b"
server_key_version: "encryptionkey@0"
I20260808 13:01:45.972077 20394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "c436192bb059470194f6d744dd84c0e1"
format_stamp: "Formatted at 2026-08-08 13:01:45 on dist-test-slave-tk9z"
server_key: "c1f81a0ef677d1105fe10b24ab316531"
server_key_iv: "658398722354004b36fc50bfb068c98b"
server_key_version: "encryptionkey@0"
I20260808 13:01:45.977877 20394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 13:01:45.982504 20417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:45.984066 20394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 13:01:45.984227 20394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "c436192bb059470194f6d744dd84c0e1"
format_stamp: "Formatted at 2026-08-08 13:01:45 on dist-test-slave-tk9z"
server_key: "c1f81a0ef677d1105fe10b24ab316531"
server_key_iv: "658398722354004b36fc50bfb068c98b"
server_key_version: "encryptionkey@0"
I20260808 13:01:45.984478 20394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:46.009521 20394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:46.010500 20394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:46.010792 20394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:46.011796 20394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:46.013332 20394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:46.013422 20394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:46.013509 20394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:46.013563 20394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:46.047637 20394 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:40281
I20260808 13:01:46.047735 20596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:40281 every 8 connection(s)
I20260808 13:01:46.049117 20394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 13:01:46.053794 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 20394
I20260808 13:01:46.054004 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 13:01:46.054526 25244 external_mini_cluster.cc:1428] Setting key ebd23024dc5dfb3a75cb210e811b4f1b
I20260808 13:01:46.057852 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44325
--builtin_ntp_servers=127.24.167.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 13:01:46.063198 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.051808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:60343 (local address 127.24.167.62:44325)
0808 13:01:46.052003 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:46.052008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:46.053455 (+ 1447us) server_negotiation.cc:407] Connection header received
0808 13:01:46.055577 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:46.055583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:46.055669 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:46.055759 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:46.057212 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.058314 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.059945 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.060651 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.060801 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.062183 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:46.062191 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:46.062198 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:46.062226 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:46.062272 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:46.062292 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:46.062420 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:46.062686 (+ 266us) server_negotiation.cc:299] Negotiation successful
0808 13:01:46.062806 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 13:01:46.064122 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.052288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:44325 (local address 127.24.167.2:60343)
0808 13:01:46.053222 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:46.053265 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:46.055297 (+ 2032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:46.056062 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:46.056073 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:46.056190 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:46.057031 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:46.057053 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.058515 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.058519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:46.059784 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:46.059797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.060875 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.060880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:46.061098 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.061837 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 13:01:46.061840 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:46.061883 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 13:01:46.061972 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:46.062439 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:46.062467 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:46.062473 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 13:01:46.062644 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 13:01:46.062969 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":290,"threads_started":1}
I20260808 13:01:46.065675 20599 heartbeater.cc:347] Connected to a master server at 127.24.167.62:44325
I20260808 13:01:46.066025 20599 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:46.066992 20599 heartbeater.cc:511] Master 127.24.167.62:44325 requested a full tablet report, sending...
I20260808 13:01:46.068717 20086 ts_manager.cc:194] Registered new tserver with Master: c436192bb059470194f6d744dd84c0e1 (127.24.167.2:40281)
I20260808 13:01:46.069741 20086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.167.2:60343
W20260808 13:01:46.412462 20604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:46.412781 20604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:46.412955 20604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:46.420513 20604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:46.420629 20604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:46.420817 20604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 13:01:46.430835 20604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:46.432317 20604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:46.434484 20604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:46.448473 20613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:46.448547 20614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:46.448838 20616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:46.449919 20604 server_base.cc:1034] running on GCE node
I20260808 13:01:46.450758 20604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:46.451786 20604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:46.453001 20604 hybrid_clock.cc:648] HybridClock initialized: now 1786194106452967 us; error 38 us; skew 500 ppm
I20260808 13:01:46.453338 20604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 13:01:46.455746 20604 webserver.cc:460] Webserver started at http://127.24.167.3:40757/ using document root <none> and password file <none>
I20260808 13:01:46.456501 20604 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:46.456626 20604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:46.456972 20604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:46.460192 20604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "0bd5b382c8ca47e3a9709389487324c3"
format_stamp: "Formatted at 2026-08-08 13:01:46 on dist-test-slave-tk9z"
server_key: "1519d54b5119864506624b92a886440f"
server_key_iv: "6546ad0aff480682a498925ac0afe806"
server_key_version: "encryptionkey@0"
I20260808 13:01:46.461139 20604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "0bd5b382c8ca47e3a9709389487324c3"
format_stamp: "Formatted at 2026-08-08 13:01:46 on dist-test-slave-tk9z"
server_key: "1519d54b5119864506624b92a886440f"
server_key_iv: "6546ad0aff480682a498925ac0afe806"
server_key_version: "encryptionkey@0"
I20260808 13:01:46.466590 20604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 13:01:46.470544 20624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:46.472007 20604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 13:01:46.472213 20604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "0bd5b382c8ca47e3a9709389487324c3"
format_stamp: "Formatted at 2026-08-08 13:01:46 on dist-test-slave-tk9z"
server_key: "1519d54b5119864506624b92a886440f"
server_key_iv: "6546ad0aff480682a498925ac0afe806"
server_key_version: "encryptionkey@0"
I20260808 13:01:46.472462 20604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:46.495570 20604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:46.496460 20604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:46.496803 20604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:46.497833 20604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:46.499698 20604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:46.499822 20604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:46.499954 20604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:46.500046 20604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:46.534446 20604 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:43777
I20260808 13:01:46.534585 20808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:43777 every 8 connection(s)
I20260808 13:01:46.535835 20604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 13:01:46.539981 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 20604
I20260808 13:01:46.540194 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 13:01:46.540856 25244 external_mini_cluster.cc:1428] Setting key 3f33ff617b33ac6f2c4861b882ac6e25
I20260808 13:01:46.549829 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.538501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:48967 (local address 127.24.167.62:44325)
0808 13:01:46.538724 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:46.538731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:46.540416 (+ 1685us) server_negotiation.cc:407] Connection header received
0808 13:01:46.542795 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:46.542801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:46.542921 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:46.543012 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:46.544351 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.545103 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.546823 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.547289 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.547391 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.548961 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:46.548965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:46.548968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:46.548986 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:46.549019 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:46.549030 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:46.549105 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:46.549393 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 13:01:46.549499 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 13:01:46.550944 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.539095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:44325 (local address 127.24.167.3:48967)
0808 13:01:46.540224 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:46.540276 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:46.542462 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:46.543272 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:46.543281 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:46.543378 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:46.544182 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:46.544210 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.545224 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.545228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:46.546668 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:46.546684 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.547496 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.547501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:46.547707 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.548595 (+ 888us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 13:01:46.548598 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:46.548641 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 13:01:46.548735 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:46.549151 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:46.549162 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:46.549169 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 13:01:46.549352 (+ 183us) client_negotiation.cc:240] Negotiation successful
0808 13:01:46.549747 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":287,"threads_started":1}
I20260808 13:01:46.552683 20809 heartbeater.cc:347] Connected to a master server at 127.24.167.62:44325
I20260808 13:01:46.553045 20809 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:46.553949 20809 heartbeater.cc:511] Master 127.24.167.62:44325 requested a full tablet report, sending...
I20260808 13:01:46.555819 20086 ts_manager.cc:194] Registered new tserver with Master: 0bd5b382c8ca47e3a9709389487324c3 (127.24.167.3:43777)
I20260808 13:01:46.556794 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 13:01:46.556806 20086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.167.3:48967
I20260808 13:01:46.567852 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.561580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.24.167.62:44325)
0808 13:01:46.561816 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:46.561825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:46.562181 (+ 356us) server_negotiation.cc:407] Connection header received
0808 13:01:46.562458 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:46.562463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:46.562533 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:46.562643 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:46.563166 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.563875 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.565372 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.565801 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.565950 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.567118 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:46.567129 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:46.567132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:46.567150 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:46.567184 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:46.567196 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:46.567293 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:46.567438 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 13:01:46.567539 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 13:01:46.574748 20389 heartbeater.cc:503] Master 127.24.167.62:44325 was elected leader, sending a full tablet report...
I20260808 13:01:46.571774 20086 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 13:01:46.579538 20086 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 13:01:46.604291 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.592544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:45485 (local address 127.0.0.1:34910)
0808 13:01:46.593442 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:46.593495 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:46.593676 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:46.595075 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:46.595084 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:46.595124 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:46.595568 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:46.595585 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.598250 (+ 2665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.598261 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:46.601212 (+ 2951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:46.601232 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.603209 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.603216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:46.603441 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.603501 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 13:01:46.603673 (+ 172us) client_negotiation.cc:240] Negotiation successful
0808 13:01:46.603895 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":351,"threads_started":1}
I20260808 13:01:46.604317 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.592837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34910 (local address 127.24.167.1:45485)
0808 13:01:46.593666 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:46.593677 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:46.593717 (+ 40us) server_negotiation.cc:407] Connection header received
0808 13:01:46.593948 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:46.593960 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:46.594299 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:46.594596 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:46.595826 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.598039 (+ 2213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.601472 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.602916 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.603071 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.603379 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:46.603744 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 13:01:46.603945 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":469,"threads_started":1}
I20260808 13:01:46.608309 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.593831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:43777 (local address 127.0.0.1:36618)
0808 13:01:46.594415 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:46.594435 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:46.594574 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:46.600910 (+ 6336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:46.600915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:46.600940 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:46.601316 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:46.601325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.603529 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.603534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:46.605598 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:46.605616 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.607452 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.607458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:46.607629 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.607661 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 13:01:46.607771 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 13:01:46.607925 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":1,"thread_start_us":274,"threads_started":1}
I20260808 13:01:46.608341 20836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.594204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.24.167.3:43777)
0808 13:01:46.600285 (+ 6081us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:46.600294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:46.600329 (+ 35us) server_negotiation.cc:407] Connection header received
0808 13:01:46.600427 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:46.600434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:46.600687 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:46.601531 (+ 844us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:46.601834 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.603399 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.605806 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.607232 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.607388 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.607662 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:46.607821 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 13:01:46.607985 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5896,"thread_start_us":5732,"threads_started":1}
I20260808 13:01:46.609027 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.594925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.24.167.2:40281)
0808 13:01:46.595652 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:46.595665 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:46.599222 (+ 3557us) server_negotiation.cc:407] Connection header received
0808 13:01:46.599331 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:46.599339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:46.599624 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:46.599902 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:46.600930 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.602591 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.605252 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.607294 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.607459 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.607709 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:46.608606 (+ 897us) server_negotiation.cc:299] Negotiation successful
0808 13:01:46.608773 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":500,"threads_started":1}
I20260808 13:01:46.609527 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.595311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:40281 (local address 127.0.0.1:57870)
0808 13:01:46.598906 (+ 3595us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:46.598923 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:46.599079 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:46.600213 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:46.600218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:46.600246 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:46.600683 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:46.600698 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.602759 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.602765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:46.604996 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:46.605020 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.607840 (+ 2820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.607846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:46.608029 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.608070 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 13:01:46.608156 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 13:01:46.608301 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3455,"mutex_wait_us":4,"thread_start_us":3298,"threads_started":1}
I20260808 13:01:46.608943 20285 tablet_service.cc:1467] Processing CreateTablet for tablet 729e402c4f6547f4af04650d559b67e4 (DEFAULT_TABLE table=test-table [id=151072decc1b4e17ad642cca28d9da6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:46.610693 20285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 729e402c4f6547f4af04650d559b67e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:46.613749 20503 tablet_service.cc:1467] Processing CreateTablet for tablet 729e402c4f6547f4af04650d559b67e4 (DEFAULT_TABLE table=test-table [id=151072decc1b4e17ad642cca28d9da6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:46.615357 20503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 729e402c4f6547f4af04650d559b67e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:46.618347 20701 tablet_service.cc:1467] Processing CreateTablet for tablet 729e402c4f6547f4af04650d559b67e4 (DEFAULT_TABLE table=test-table [id=151072decc1b4e17ad642cca28d9da6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:46.619861 20701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 729e402c4f6547f4af04650d559b67e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:46.632776 20839 tablet_bootstrap.cc:492] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1: Bootstrap starting.
I20260808 13:01:46.633976 20840 tablet_bootstrap.cc:492] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf: Bootstrap starting.
I20260808 13:01:46.635078 20841 tablet_bootstrap.cc:492] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3: Bootstrap starting.
I20260808 13:01:46.638103 20841 tablet_bootstrap.cc:654] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:46.638701 20839 tablet_bootstrap.cc:654] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:46.639065 20840 tablet_bootstrap.cc:654] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:46.639434 20841 log.cc:824] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:46.640110 20839 log.cc:824] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:46.640367 20840 log.cc:824] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:46.643364 20839 tablet_bootstrap.cc:492] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1: No bootstrap required, opened a new log
I20260808 13:01:46.643577 20841 tablet_bootstrap.cc:492] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3: No bootstrap required, opened a new log
I20260808 13:01:46.643689 20839 ts_tablet_manager.cc:1397] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260808 13:01:46.643895 20841 ts_tablet_manager.cc:1397] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260808 13:01:46.643955 20840 tablet_bootstrap.cc:492] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf: No bootstrap required, opened a new log
I20260808 13:01:46.644217 20840 ts_tablet_manager.cc:1397] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260808 13:01:46.652676 20839 raft_consensus.cc:348] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7207bc99078144958504f9069c6261cf" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45485 } } peers { permanent_uuid: "0bd5b382c8ca47e3a9709389487324c3" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43777 } } peers { permanent_uuid: "c436192bb059470194f6d744dd84c0e1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40281 } }
I20260808 13:01:46.653118 20839 raft_consensus.cc:374] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:46.653236 20839 raft_consensus.cc:729] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c436192bb059470194f6d744dd84c0e1, State: Initialized, Role: FOLLOWER
I20260808 13:01:46.653955 20839 consensus_queue.cc:260] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7207bc99078144958504f9069c6261cf" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45485 } } peers { permanent_uuid: "0bd5b382c8ca47e3a9709389487324c3" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43777 } } peers { permanent_uuid: "c436192bb059470194f6d744dd84c0e1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40281 } }
I20260808 13:01:46.655262 20599 heartbeater.cc:503] Master 127.24.167.62:44325 was elected leader, sending a full tablet report...
I20260808 13:01:46.655845 20839 ts_tablet_manager.cc:1428] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260808 13:01:46.656162 20841 raft_consensus.cc:348] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7207bc99078144958504f9069c6261cf" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45485 } } peers { permanent_uuid: "0bd5b382c8ca47e3a9709389487324c3" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43777 } } peers { permanent_uuid: "c436192bb059470194f6d744dd84c0e1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40281 } }
I20260808 13:01:46.656675 20841 raft_consensus.cc:374] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:46.656795 20841 raft_consensus.cc:729] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bd5b382c8ca47e3a9709389487324c3, State: Initialized, Role: FOLLOWER
I20260808 13:01:46.657624 20841 consensus_queue.cc:260] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7207bc99078144958504f9069c6261cf" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45485 } } peers { permanent_uuid: "0bd5b382c8ca47e3a9709389487324c3" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43777 } } peers { permanent_uuid: "c436192bb059470194f6d744dd84c0e1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40281 } }
I20260808 13:01:46.658077 20840 raft_consensus.cc:348] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7207bc99078144958504f9069c6261cf" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45485 } } peers { permanent_uuid: "0bd5b382c8ca47e3a9709389487324c3" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43777 } } peers { permanent_uuid: "c436192bb059470194f6d744dd84c0e1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40281 } }
I20260808 13:01:46.659173 20840 raft_consensus.cc:374] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:46.659400 20840 raft_consensus.cc:729] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7207bc99078144958504f9069c6261cf, State: Initialized, Role: FOLLOWER
I20260808 13:01:46.660140 20809 heartbeater.cc:503] Master 127.24.167.62:44325 was elected leader, sending a full tablet report...
I20260808 13:01:46.660996 20841 ts_tablet_manager.cc:1428] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3: Time spent starting tablet: real 0.017s user 0.011s sys 0.006s
I20260808 13:01:46.661718 20840 consensus_queue.cc:260] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7207bc99078144958504f9069c6261cf" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45485 } } peers { permanent_uuid: "0bd5b382c8ca47e3a9709389487324c3" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43777 } } peers { permanent_uuid: "c436192bb059470194f6d744dd84c0e1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40281 } }
I20260808 13:01:46.663928 20840 ts_tablet_manager.cc:1428] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf: Time spent starting tablet: real 0.020s user 0.016s sys 0.000s
W20260808 13:01:46.787724 20811 tablet.cc:2369] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 13:01:46.801565 20600 tablet.cc:2369] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 13:01:46.804095 20390 tablet.cc:2369] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:01:46.827075 20849 raft_consensus.cc:482] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 13:01:46.827487 20849 raft_consensus.cc:504] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7207bc99078144958504f9069c6261cf" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45485 } } peers { permanent_uuid: "0bd5b382c8ca47e3a9709389487324c3" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43777 } } peers { permanent_uuid: "c436192bb059470194f6d744dd84c0e1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40281 } }
I20260808 13:01:46.829299 20849 leader_election.cc:288] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bd5b382c8ca47e3a9709389487324c3 (127.24.167.3:43777), c436192bb059470194f6d744dd84c0e1 (127.24.167.2:40281)
I20260808 13:01:46.837251 20836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.829770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:39731 (local address 127.24.167.3:43777)
0808 13:01:46.829981 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:46.829986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:46.830619 (+ 633us) server_negotiation.cc:407] Connection header received
0808 13:01:46.830862 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:46.830867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:46.830952 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:46.831069 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:46.831943 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.833041 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.835293 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.836242 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.836368 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.836418 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:46.836815 (+ 397us) server_negotiation.cc:299] Negotiation successful
0808 13:01:46.836983 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 13:01:46.837245 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.829739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:43777 (local address 127.24.167.1:39731)
0808 13:01:46.830492 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:46.830514 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:46.830669 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:46.831267 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:46.831273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:46.831307 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:46.831704 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:46.831718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.833191 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.833196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:46.835085 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:46.835105 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.836426 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.836433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:46.836598 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.836647 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 13:01:46.836739 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 13:01:46.836883 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":126,"thread_start_us":392,"threads_started":1}
I20260808 13:01:46.837626 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.830904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:40757 (local address 127.24.167.2:40281)
0808 13:01:46.831168 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:46.831174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:46.831529 (+ 355us) server_negotiation.cc:407] Connection header received
0808 13:01:46.831667 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:46.831674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:46.831767 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:46.831862 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:46.832638 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.833478 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.835724 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.836579 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.836754 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.836819 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:46.837169 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 13:01:46.837324 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 13:01:46.837642 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.830687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:40281 (local address 127.24.167.1:40757)
0808 13:01:46.831414 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:46.831429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:46.831544 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:46.831944 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:46.831948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:46.831973 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:46.832328 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:46.832395 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.833632 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.833636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:46.835540 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:46.835555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.836816 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.836824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:46.836989 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.837021 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 13:01:46.837119 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 13:01:46.837244 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":108,"thread_start_us":415,"threads_started":1}
I20260808 13:01:46.838959 20736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729e402c4f6547f4af04650d559b67e4" candidate_uuid: "7207bc99078144958504f9069c6261cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bd5b382c8ca47e3a9709389487324c3" is_pre_election: true
I20260808 13:01:46.838932 20531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729e402c4f6547f4af04650d559b67e4" candidate_uuid: "7207bc99078144958504f9069c6261cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c436192bb059470194f6d744dd84c0e1" is_pre_election: true
I20260808 13:01:46.839546 20531 raft_consensus.cc:2454] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7207bc99078144958504f9069c6261cf in term 0.
I20260808 13:01:46.839555 20736 raft_consensus.cc:2454] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7207bc99078144958504f9069c6261cf in term 0.
I20260808 13:01:46.840220 20203 leader_election.cc:302] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7207bc99078144958504f9069c6261cf, c436192bb059470194f6d744dd84c0e1; no voters:
I20260808 13:01:46.840628 20849 raft_consensus.cc:2781] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 13:01:46.840804 20849 raft_consensus.cc:482] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 13:01:46.840956 20849 raft_consensus.cc:3037] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:46.843282 20849 raft_consensus.cc:504] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7207bc99078144958504f9069c6261cf" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45485 } } peers { permanent_uuid: "0bd5b382c8ca47e3a9709389487324c3" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43777 } } peers { permanent_uuid: "c436192bb059470194f6d744dd84c0e1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40281 } }
I20260808 13:01:46.843890 20849 leader_election.cc:288] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [CANDIDATE]: Term 1 election: Requested vote from peers 0bd5b382c8ca47e3a9709389487324c3 (127.24.167.3:43777), c436192bb059470194f6d744dd84c0e1 (127.24.167.2:40281)
I20260808 13:01:46.844606 20531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729e402c4f6547f4af04650d559b67e4" candidate_uuid: "7207bc99078144958504f9069c6261cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c436192bb059470194f6d744dd84c0e1"
I20260808 13:01:46.844594 20736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729e402c4f6547f4af04650d559b67e4" candidate_uuid: "7207bc99078144958504f9069c6261cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bd5b382c8ca47e3a9709389487324c3"
I20260808 13:01:46.844833 20736 raft_consensus.cc:3037] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:46.844907 20531 raft_consensus.cc:3037] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:46.848368 20531 raft_consensus.cc:2454] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7207bc99078144958504f9069c6261cf in term 1.
I20260808 13:01:46.848412 20736 raft_consensus.cc:2454] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7207bc99078144958504f9069c6261cf in term 1.
I20260808 13:01:46.848891 20203 leader_election.cc:302] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7207bc99078144958504f9069c6261cf, c436192bb059470194f6d744dd84c0e1; no voters:
I20260808 13:01:46.849196 20849 raft_consensus.cc:2781] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:01:46.849711 20849 raft_consensus.cc:686] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [term 1 LEADER]: Becoming Leader. State: Replica: 7207bc99078144958504f9069c6261cf, State: Running, Role: LEADER
I20260808 13:01:46.850216 20849 consensus_queue.cc:237] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7207bc99078144958504f9069c6261cf" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45485 } } peers { permanent_uuid: "0bd5b382c8ca47e3a9709389487324c3" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43777 } } peers { permanent_uuid: "c436192bb059470194f6d744dd84c0e1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40281 } }
I20260808 13:01:46.855682 20086 catalog_manager.cc:5495] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7207bc99078144958504f9069c6261cf (127.24.167.1). New cstate: current_term: 1 leader_uuid: "7207bc99078144958504f9069c6261cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7207bc99078144958504f9069c6261cf" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 45485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bd5b382c8ca47e3a9709389487324c3" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c436192bb059470194f6d744dd84c0e1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40281 } health_report { overall_health: UNKNOWN } } }
I20260808 13:01:46.908890 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.903442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.24.167.1:45485)
0808 13:01:46.903680 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:46.903686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:46.903707 (+ 21us) server_negotiation.cc:407] Connection header received
0808 13:01:46.903835 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:46.903839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:46.903939 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:46.904108 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:01:46.904495 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.905193 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.906704 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.907130 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.907269 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.907775 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:01:46.908279 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:01:46.908458 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:46.908517 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 13:01:46.908596 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 13:01:46.920073 20531 raft_consensus.cc:1261] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1 [term 1 FOLLOWER]: Refusing update from remote peer 7207bc99078144958504f9069c6261cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:01:46.920069 20736 raft_consensus.cc:1261] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3 [term 1 FOLLOWER]: Refusing update from remote peer 7207bc99078144958504f9069c6261cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:01:46.921080 20855 consensus_queue.cc:1035] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bd5b382c8ca47e3a9709389487324c3" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:01:46.921540 20849 consensus_queue.cc:1035] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c436192bb059470194f6d744dd84c0e1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 40281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:01:46.947522 20086 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:01:46.947809 20086 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:01:46.951640 20086 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 746589a9de0f4d1cb3938dee512ac557: Sending DeleteTablet for 3 replicas of tablet 729e402c4f6547f4af04650d559b67e4
I20260808 13:01:46.952782 20285 tablet_service.cc:1514] Processing DeleteTablet for tablet 729e402c4f6547f4af04650d559b67e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:46 UTC) from {username='slave'} at 127.0.0.1:34910
I20260808 13:01:46.953015 20503 tablet_service.cc:1514] Processing DeleteTablet for tablet 729e402c4f6547f4af04650d559b67e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:46 UTC) from {username='slave'} at 127.0.0.1:57870
I20260808 13:01:46.953716 20701 tablet_service.cc:1514] Processing DeleteTablet for tablet 729e402c4f6547f4af04650d559b67e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:46 UTC) from {username='slave'} at 127.0.0.1:36618
I20260808 13:01:46.954229 20880 tablet_replica.cc:330] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1: stopping tablet replica
I20260808 13:01:46.954785 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 20180
I20260808 13:01:46.954769 20880 raft_consensus.cc:2229] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:01:46.955307 20880 raft_consensus.cc:2258] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:46.957647 20880 ts_tablet_manager.cc:1905] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:46.963892 20882 tablet_replica.cc:330] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3: stopping tablet replica
I20260808 13:01:46.964455 20882 raft_consensus.cc:2229] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:01:46.965008 20882 raft_consensus.cc:2258] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:46.965852 20879 tablet_replica.cc:330] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf: stopping tablet replica
I20260808 13:01:46.966320 20879 raft_consensus.cc:2229] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [term 1 LEADER]: Raft consensus shutting down.
I20260808 13:01:46.966403 20880 ts_tablet_manager.cc:1918] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:46.966710 20880 log.cc:1196] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/729e402c4f6547f4af04650d559b67e4
I20260808 13:01:46.967097 20879 raft_consensus.cc:2258] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:46.967998 20880 ts_tablet_manager.cc:1939] T 729e402c4f6547f4af04650d559b67e4 P c436192bb059470194f6d744dd84c0e1: Deleting consensus metadata
I20260808 13:01:46.969202 20879 ts_tablet_manager.cc:1905] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:46.969820 20882 ts_tablet_manager.cc:1905] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:46.970394 20062 catalog_manager.cc:4841] TS c436192bb059470194f6d744dd84c0e1 (127.24.167.2:40281): tablet 729e402c4f6547f4af04650d559b67e4 (table test-table [id=151072decc1b4e17ad642cca28d9da6e]) successfully deleted
I20260808 13:01:46.976362 20882 ts_tablet_manager.cc:1918] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:46.976629 20882 log.cc:1196] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/729e402c4f6547f4af04650d559b67e4
I20260808 13:01:46.977291 20882 ts_tablet_manager.cc:1939] T 729e402c4f6547f4af04650d559b67e4 P 0bd5b382c8ca47e3a9709389487324c3: Deleting consensus metadata
I20260808 13:01:46.978621 20879 ts_tablet_manager.cc:1918] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:46.978813 20879 log.cc:1196] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/729e402c4f6547f4af04650d559b67e4
I20260808 13:01:46.979492 20879 ts_tablet_manager.cc:1939] T 729e402c4f6547f4af04650d559b67e4 P 7207bc99078144958504f9069c6261cf: Deleting consensus metadata
I20260808 13:01:46.979769 20062 catalog_manager.cc:4841] TS 0bd5b382c8ca47e3a9709389487324c3 (127.24.167.3:43777): tablet 729e402c4f6547f4af04650d559b67e4 (table test-table [id=151072decc1b4e17ad642cca28d9da6e]) successfully deleted
I20260808 13:01:46.981091 20062 catalog_manager.cc:4841] TS 7207bc99078144958504f9069c6261cf (127.24.167.1:45485): tablet 729e402c4f6547f4af04650d559b67e4 (table test-table [id=151072decc1b4e17ad642cca28d9da6e]) successfully deleted
I20260808 13:01:46.986315 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:46.956437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.24.167.1:45485)
0808 13:01:46.956625 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:46.956631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:46.974981 (+ 18350us) server_negotiation.cc:407] Connection header received
0808 13:01:46.975293 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:46.975299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:46.975398 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:46.975482 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:46.976319 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.979389 (+ 3070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.982493 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:46.983147 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:46.983303 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:46.984609 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:46.984620 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:46.984639 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:46.985476 (+ 837us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:46.985537 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:46.985561 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:46.985668 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:46.985850 (+ 182us) server_negotiation.cc:299] Negotiation successful
0808 13:01:46.985959 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 13:01:46.987098 20381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:47.088465 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 20180
I20260808 13:01:47.116791 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 20394
I20260808 13:01:47.124100 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:47.117706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57884 (local address 127.24.167.2:40281)
0808 13:01:47.117894 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:47.117899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:47.117931 (+ 32us) server_negotiation.cc:407] Connection header received
0808 13:01:47.117992 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:47.117996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:47.118095 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:47.118193 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:47.118730 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:47.119505 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:47.120826 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:47.121250 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:47.121341 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:47.122520 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:47.122535 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:47.122559 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:47.123290 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:47.123342 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:47.123362 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:47.123461 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:47.123699 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 13:01:47.123802 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 13:01:47.124867 20592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:47.214493 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 20394
I20260808 13:01:47.238075 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 20604
I20260808 13:01:47.245241 20836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:47.239000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.24.167.3:43777)
0808 13:01:47.239203 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:47.239209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:47.239231 (+ 22us) server_negotiation.cc:407] Connection header received
0808 13:01:47.239408 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:47.239412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:47.239507 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:47.239617 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:47.240065 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:47.240781 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:47.242070 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:47.242593 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:47.242737 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:47.243833 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:47.243842 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:47.243860 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:47.244458 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:47.244509 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:47.244530 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:47.244664 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:47.244805 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 13:01:47.244919 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 13:01:47.245936 20800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:47.334926 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 20604
I20260808 13:01:47.359383 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 20034
I20260808 13:01:47.360522 20134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:47.473377 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 20034
2026-08-08T13:01:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3052 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:01:47 dist-test-slave-tk9z krb5kdc[20896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:01:47 dist-test-slave-tk9z krb5kdc[20896](info): set up 2 sockets
Aug 08 13:01:47 dist-test-slave-tk9z krb5kdc[20896](info): commencing operation
krb5kdc: starting...
W20260808 13:01:48.822880 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.294s user 0.003s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:01:49 dist-test-slave-tk9z krb5kdc[20896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:01:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:01:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 13:01:49.139810 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:42273
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:42273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:01:49.495297 20913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:49.495651 20913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:49.495837 20913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:49.503166 20913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:01:49.503319 20913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:49.503439 20913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:49.503546 20913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:01:49.503633 20913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:01:49.513635 20913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:42273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:42273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:49.515024 20913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:49.516362 20913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:49.528388 20921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:49.528425 20926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:49.528455 20924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:49.530125 20913 server_base.cc:1034] running on GCE node
I20260808 13:01:49.531059 20913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:49.532536 20913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:49.533748 20913 hybrid_clock.cc:648] HybridClock initialized: now 1786194109533702 us; error 36 us; skew 500 ppm
I20260808 13:01:49.534123 20913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:49 dist-test-slave-tk9z krb5kdc[20896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194109, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:49.537364 20913 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:01:49.538987 20913 webserver.cc:460] Webserver started at http://127.24.167.62:35049/ using document root <none> and password file <none>
I20260808 13:01:49.539726 20913 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:49.539876 20913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:49.540201 20913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:49.543485 20913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "4393e033aa9a4b50937d04485e3f873a"
format_stamp: "Formatted at 2026-08-08 13:01:49 on dist-test-slave-tk9z"
server_key: "7326d969bca35f9f332fc92a4ac8b541"
server_key_iv: "64dcd88616a8b66e078b61dc47a1b354"
server_key_version: "encryptionkey@0"
I20260808 13:01:49.544317 20913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "4393e033aa9a4b50937d04485e3f873a"
format_stamp: "Formatted at 2026-08-08 13:01:49 on dist-test-slave-tk9z"
server_key: "7326d969bca35f9f332fc92a4ac8b541"
server_key_iv: "64dcd88616a8b66e078b61dc47a1b354"
server_key_version: "encryptionkey@0"
I20260808 13:01:49.549732 20913 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 13:01:49.553373 20936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:49.554764 20913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:01:49.554963 20913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "4393e033aa9a4b50937d04485e3f873a"
format_stamp: "Formatted at 2026-08-08 13:01:49 on dist-test-slave-tk9z"
server_key: "7326d969bca35f9f332fc92a4ac8b541"
server_key_iv: "64dcd88616a8b66e078b61dc47a1b354"
server_key_version: "encryptionkey@0"
I20260808 13:01:49.555171 20913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:49.578589 20913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:49.583223 20913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:49.583577 20913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:49.603365 20913 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:42273
I20260808 13:01:49.603427 21006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:42273 every 8 connection(s)
I20260808 13:01:49.604802 20913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:01:49.608834 21007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:49.611115 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 20913
I20260808 13:01:49.611334 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:01:49.611783 25244 external_mini_cluster.cc:1428] Setting key 590cf343968975b51905e30060e29f6b
I20260808 13:01:49.621615 21007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a: Bootstrap starting.
I20260808 13:01:49.625850 21007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:49.627156 21007 log.cc:824] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a: Log is configured to *not* fsync() on all Append() calls
Aug 08 13:01:49 dist-test-slave-tk9z krb5kdc[20896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:01:49.630112 21007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a: No bootstrap required, opened a new log
I20260808 13:01:49.636013 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:49.613705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.24.167.62:42273)
0808 13:01:49.614596 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:49.614611 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:49.614656 (+ 45us) server_negotiation.cc:407] Connection header received
0808 13:01:49.616351 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:49.616373 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:49.616752 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:49.617056 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:49.619009 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:49.620367 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:49.622264 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:49.622986 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:49.623139 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:49.628524 (+ 5385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:49.628539 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:49.628565 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:49.628603 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:49.631767 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:49.632525 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:49.632540 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:49.632546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:49.632721 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:49.633134 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:49.633141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:49.633149 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:49.633703 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:49.634028 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:49.634545 (+ 517us) server_negotiation.cc:299] Negotiation successful
0808 13:01:49.634982 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":403,"threads_started":1}
I20260808 13:01:49.638777 21007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4393e033aa9a4b50937d04485e3f873a" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42273 } }
I20260808 13:01:49.639257 21007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:49.639370 21007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4393e033aa9a4b50937d04485e3f873a, State: Initialized, Role: FOLLOWER
I20260808 13:01:49.639907 21007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4393e033aa9a4b50937d04485e3f873a" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42273 } }
I20260808 13:01:49.640125 21007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:01:49.640220 21007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:01:49.640347 21007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:49.642542 21007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4393e033aa9a4b50937d04485e3f873a" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42273 } }
I20260808 13:01:49.643101 21007 leader_election.cc:302] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4393e033aa9a4b50937d04485e3f873a; no voters:
I20260808 13:01:49.643698 21007 leader_election.cc:288] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:01:49.643805 21013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:01:49.645058 21013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [term 1 LEADER]: Becoming Leader. State: Replica: 4393e033aa9a4b50937d04485e3f873a, State: Running, Role: LEADER
I20260808 13:01:49.645624 21013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4393e033aa9a4b50937d04485e3f873a" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42273 } }
I20260808 13:01:49.645959 21007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:01:49.648661 21016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4393e033aa9a4b50937d04485e3f873a. Latest consensus state: current_term: 1 leader_uuid: "4393e033aa9a4b50937d04485e3f873a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4393e033aa9a4b50937d04485e3f873a" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42273 } } }
I20260808 13:01:49.648762 21014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4393e033aa9a4b50937d04485e3f873a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4393e033aa9a4b50937d04485e3f873a" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 42273 } } }
I20260808 13:01:49.649070 21016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [sys.catalog]: This master's current role is: LEADER
I20260808 13:01:49.649190 21014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a [sys.catalog]: This master's current role is: LEADER
I20260808 13:01:49.654057 21027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:01:49.660043 21027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:01:49.669225 21027 catalog_manager.cc:1347] Generated new cluster ID: 161a67c468fa421eae2dc4249efeda59
I20260808 13:01:49.669344 21027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:01:49.682547 21027 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:01:49.684387 21027 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:01:49.692042 21027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a: Generated new TSK 0
I20260808 13:01:49.693034 21027 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 13:01:49.767308 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42273
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:01:50.126508 21042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:50.126883 21042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:50.126993 21042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:50.134609 21042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:50.134743 21042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:50.134861 21042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 13:01:50.145439 21042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:50.147053 21042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:50.148588 21042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:50.162520 21054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:50.162436 21051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:50.162534 21050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:50.163918 21042 server_base.cc:1034] running on GCE node
I20260808 13:01:50.164587 21042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:50.165588 21042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:50.166826 21042 hybrid_clock.cc:648] HybridClock initialized: now 1786194110166776 us; error 58 us; skew 500 ppm
I20260808 13:01:50.167160 21042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:50 dist-test-slave-tk9z krb5kdc[20896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194110, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:50.175199 21042 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 13:01:50.176748 21042 webserver.cc:460] Webserver started at http://127.24.167.1:46785/ using document root <none> and password file <none>
I20260808 13:01:50.177443 21042 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:50.177552 21042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:50.177830 21042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:50.180938 21042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "59156f0786644e85a60f2bb82f6df561"
format_stamp: "Formatted at 2026-08-08 13:01:50 on dist-test-slave-tk9z"
server_key: "97c15f920a85d85f9b8ce55d3f5e8dc2"
server_key_iv: "5114f4ce0760924acef6ebbde04580b7"
server_key_version: "encryptionkey@0"
I20260808 13:01:50.181710 21042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "59156f0786644e85a60f2bb82f6df561"
format_stamp: "Formatted at 2026-08-08 13:01:50 on dist-test-slave-tk9z"
server_key: "97c15f920a85d85f9b8ce55d3f5e8dc2"
server_key_iv: "5114f4ce0760924acef6ebbde04580b7"
server_key_version: "encryptionkey@0"
I20260808 13:01:50.187179 21042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 13:01:50.190876 21068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:50.192543 21042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 13:01:50.192730 21042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "59156f0786644e85a60f2bb82f6df561"
format_stamp: "Formatted at 2026-08-08 13:01:50 on dist-test-slave-tk9z"
server_key: "97c15f920a85d85f9b8ce55d3f5e8dc2"
server_key_iv: "5114f4ce0760924acef6ebbde04580b7"
server_key_version: "encryptionkey@0"
I20260808 13:01:50.192945 21042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:50.212142 21042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:50.216675 21042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:50.216996 21042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:50.218091 21042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:50.219666 21042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:50.219775 21042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:50.219930 21042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:50.220005 21042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:50.253928 21042 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:34957
I20260808 13:01:50.253975 21239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:34957 every 8 connection(s)
I20260808 13:01:50.255488 21042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 13:01:50.260197 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 21042
I20260808 13:01:50.260404 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 13:01:50.260898 25244 external_mini_cluster.cc:1428] Setting key bdeb75b820aff275b1a6cf771574a7e8
Aug 08 13:01:50 dist-test-slave-tk9z krb5kdc[20896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194110, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:01:50.278110 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:50.258084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:35937 (local address 127.24.167.62:42273)
0808 13:01:50.258661 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:50.258667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:50.259648 (+ 981us) server_negotiation.cc:407] Connection header received
0808 13:01:50.261802 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:50.261808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:50.261893 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:50.262011 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:50.263913 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:50.265122 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:50.266745 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:50.267482 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:50.267640 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:50.271284 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:50.271302 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:50.271310 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:50.271357 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:50.275016 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:50.275757 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:50.275762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:50.275764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:50.275886 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:50.276384 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:50.276387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:50.276389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:50.276644 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:50.276759 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:50.277661 (+ 902us) server_negotiation.cc:299] Negotiation successful
0808 13:01:50.277835 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":295,"threads_started":1}
I20260808 13:01:50.279680 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:50.258674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42273 (local address 127.24.167.1:35937)
0808 13:01:50.259429 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:50.259482 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:50.261491 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:50.262297 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:50.262308 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:50.263142 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:50.263728 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:50.263746 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:50.265297 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:50.265301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:50.266600 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:50.266612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:50.267702 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:50.267707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:50.267905 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:50.268587 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:01:50.268616 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:50.271038 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:50.275163 (+ 4125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:50.275169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:50.275181 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:50.275584 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:50.275998 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:50.276005 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:50.276010 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:50.276266 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:50.276774 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:50.276784 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:50.277265 (+ 481us) client_negotiation.cc:769] Sending connection context
0808 13:01:50.277568 (+ 303us) client_negotiation.cc:240] Negotiation successful
0808 13:01:50.278160 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":294,"threads_started":1}
I20260808 13:01:50.281406 21240 heartbeater.cc:347] Connected to a master server at 127.24.167.62:42273
I20260808 13:01:50.281737 21240 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:50.282580 21240 heartbeater.cc:511] Master 127.24.167.62:42273 requested a full tablet report, sending...
I20260808 13:01:50.284747 20960 ts_manager.cc:194] Registered new tserver with Master: 59156f0786644e85a60f2bb82f6df561 (127.24.167.1:34957)
I20260808 13:01:50.286208 20960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:35937
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 13:01:50.329027 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42273
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:01:50.673385 21250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:50.673664 21250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:50.673884 21250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:50.681182 21250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:50.681314 21250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:50.681504 21250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 13:01:50.691485 21250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:50.692854 21250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:50.694451 21250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:50.707947 21257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:50.707970 21260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:50.708081 21258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:50.709693 21250 server_base.cc:1034] running on GCE node
I20260808 13:01:50.710314 21250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:50.711328 21250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:50.712529 21250 hybrid_clock.cc:648] HybridClock initialized: now 1786194110712481 us; error 41 us; skew 500 ppm
I20260808 13:01:50.712848 21250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:50 dist-test-slave-tk9z krb5kdc[20896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194110, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:50.716043 21250 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 13:01:50.717487 21250 webserver.cc:460] Webserver started at http://127.24.167.2:42031/ using document root <none> and password file <none>
I20260808 13:01:50.718171 21250 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:50.718302 21250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:50.718680 21250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:50.721602 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "6148332e10524fd686375a98516bb306"
format_stamp: "Formatted at 2026-08-08 13:01:50 on dist-test-slave-tk9z"
server_key: "cd3445d40ff9235fe122dcbc3de3da7d"
server_key_iv: "77934561d263a9f1dde5cdef4f34e921"
server_key_version: "encryptionkey@0"
I20260808 13:01:50.722438 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "6148332e10524fd686375a98516bb306"
format_stamp: "Formatted at 2026-08-08 13:01:50 on dist-test-slave-tk9z"
server_key: "cd3445d40ff9235fe122dcbc3de3da7d"
server_key_iv: "77934561d263a9f1dde5cdef4f34e921"
server_key_version: "encryptionkey@0"
I20260808 13:01:50.727732 21250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 13:01:50.731300 21269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:50.732574 21250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 13:01:50.732738 21250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "6148332e10524fd686375a98516bb306"
format_stamp: "Formatted at 2026-08-08 13:01:50 on dist-test-slave-tk9z"
server_key: "cd3445d40ff9235fe122dcbc3de3da7d"
server_key_iv: "77934561d263a9f1dde5cdef4f34e921"
server_key_version: "encryptionkey@0"
I20260808 13:01:50.732939 21250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:50.766623 21250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:50.771185 21250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:50.771505 21250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:50.772514 21250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:50.773933 21250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:50.774009 21250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:50.774176 21250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:50.774232 21250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:50.808339 21250 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:35817
I20260808 13:01:50.808390 21439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:35817 every 8 connection(s)
I20260808 13:01:50.809669 21250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 13:01:50.819978 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 21250
I20260808 13:01:50.820194 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 13:01:50.820755 25244 external_mini_cluster.cc:1428] Setting key e71e6ffe25d30975cb08f69617c9f057
Aug 08 13:01:50 dist-test-slave-tk9z krb5kdc[20896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194110, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:01:50.831656 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:50.812403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:58659 (local address 127.24.167.62:42273)
0808 13:01:50.812876 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:50.812883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:50.813933 (+ 1050us) server_negotiation.cc:407] Connection header received
0808 13:01:50.815785 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:50.815790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:50.815860 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:50.815962 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:50.817454 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:50.818152 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:50.819917 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:50.820429 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:50.820527 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:50.824218 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:50.824228 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:50.824236 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:50.824283 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:50.827883 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:50.828782 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:50.828790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:50.828797 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:50.828972 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:50.829467 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:50.829474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:50.829479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:50.829866 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:50.830029 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:50.830881 (+ 852us) server_negotiation.cc:299] Negotiation successful
0808 13:01:50.831144 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":219,"threads_started":1}
I20260808 13:01:50.832583 21443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:50.813071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42273 (local address 127.24.167.2:58659)
0808 13:01:50.813761 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:50.813806 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:50.815526 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:50.816172 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:50.816180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:50.816726 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:50.817290 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:50.817307 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:50.818287 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:50.818291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:50.819773 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:50.819790 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:50.820663 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:50.820671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:50.820968 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:50.821730 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:01:50.821752 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:50.823979 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:50.828031 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:50.828042 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:50.828060 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:50.828637 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:50.829088 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:50.829095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:50.829101 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:50.829335 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:50.830006 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:50.830015 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:50.830459 (+ 444us) client_negotiation.cc:769] Sending connection context
0808 13:01:50.830736 (+ 277us) client_negotiation.cc:240] Negotiation successful
0808 13:01:50.831270 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":286,"threads_started":1}
I20260808 13:01:50.834050 21440 heartbeater.cc:347] Connected to a master server at 127.24.167.62:42273
I20260808 13:01:50.834424 21440 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:50.835304 21440 heartbeater.cc:511] Master 127.24.167.62:42273 requested a full tablet report, sending...
I20260808 13:01:50.837265 20960 ts_manager.cc:194] Registered new tserver with Master: 6148332e10524fd686375a98516bb306 (127.24.167.2:35817)
I20260808 13:01:50.838510 20960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:58659
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 13:01:50.887904 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42273
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:01:51.248523 21447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:51.248863 21447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:51.249037 21447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:51.256642 21447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:51.256793 21447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:51.257018 21447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 13:01:51.267308 21447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:42273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:51.268841 21447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:51.270283 21447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:51.283344 21452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:51.283418 21459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:51.283622 21453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:51.284497 21447 server_base.cc:1034] running on GCE node
I20260808 13:01:51.285183 21447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:51.286162 21447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:51.287391 21447 hybrid_clock.cc:648] HybridClock initialized: now 1786194111287321 us; error 77 us; skew 500 ppm
I20260808 13:01:51.287719 21447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 13:01:51.289186 21240 heartbeater.cc:503] Master 127.24.167.62:42273 was elected leader, sending a full tablet report...
Aug 08 13:01:51 dist-test-slave-tk9z krb5kdc[20896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194111, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:51.291344 21447 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 13:01:51.292807 21447 webserver.cc:460] Webserver started at http://127.24.167.3:34777/ using document root <none> and password file <none>
I20260808 13:01:51.293511 21447 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:51.293648 21447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:51.293977 21447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:51.296943 21447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "7b177b0128dc4d858a63a991a377cb1f"
format_stamp: "Formatted at 2026-08-08 13:01:51 on dist-test-slave-tk9z"
server_key: "d654df6743d561a238fc596a1fd9fe5b"
server_key_iv: "4886fc31406d4ab3290ea5de27e95478"
server_key_version: "encryptionkey@0"
I20260808 13:01:51.297729 21447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "7b177b0128dc4d858a63a991a377cb1f"
format_stamp: "Formatted at 2026-08-08 13:01:51 on dist-test-slave-tk9z"
server_key: "d654df6743d561a238fc596a1fd9fe5b"
server_key_iv: "4886fc31406d4ab3290ea5de27e95478"
server_key_version: "encryptionkey@0"
I20260808 13:01:51.303170 21447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 13:01:51.306718 21474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:51.308043 21447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 13:01:51.308223 21447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "7b177b0128dc4d858a63a991a377cb1f"
format_stamp: "Formatted at 2026-08-08 13:01:51 on dist-test-slave-tk9z"
server_key: "d654df6743d561a238fc596a1fd9fe5b"
server_key_iv: "4886fc31406d4ab3290ea5de27e95478"
server_key_version: "encryptionkey@0"
I20260808 13:01:51.308429 21447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:51.328541 21447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:51.333010 21447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:51.333333 21447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:51.334300 21447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:51.335655 21447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:51.335758 21447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:51.335903 21447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:51.335973 21447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:51.369904 21447 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:38811
I20260808 13:01:51.369946 21644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:38811 every 8 connection(s)
I20260808 13:01:51.371312 21447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 13:01:51.380688 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 21447
I20260808 13:01:51.380884 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 13:01:51.381528 25244 external_mini_cluster.cc:1428] Setting key fc7ef54d69ff4b8812d6734035f3d471
Aug 08 13:01:51 dist-test-slave-tk9z krb5kdc[20896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194111, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:01:51.392474 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.374006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:57689 (local address 127.24.167.62:42273)
0808 13:01:51.374510 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:51.374514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:51.375471 (+ 957us) server_negotiation.cc:407] Connection header received
0808 13:01:51.377260 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:51.377265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:51.377350 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:51.377453 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:51.379040 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.379818 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.382107 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.382695 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.382828 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.387035 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:51.387041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:51.387044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:51.387080 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:51.389632 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:51.390358 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:51.390362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:51.390364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:51.390538 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:51.390949 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:51.390953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:51.390955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:51.391218 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:51.391343 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:51.392022 (+ 679us) server_negotiation.cc:299] Negotiation successful
0808 13:01:51.392203 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":208,"threads_started":1}
I20260808 13:01:51.393692 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.374599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:42273 (local address 127.24.167.3:57689)
0808 13:01:51.375290 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:51.375343 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:51.377003 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:51.377666 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:51.377674 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:51.378196 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:51.378887 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.378905 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.379954 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.379958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.381933 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.381949 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.383033 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.383040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.383333 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.384219 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:01:51.384249 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:51.386816 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:51.389770 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:51.389777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:51.389790 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:51.390226 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:51.390636 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:51.390640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:51.390642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:51.390839 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:51.391326 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:51.391332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:51.391673 (+ 341us) client_negotiation.cc:769] Sending connection context
0808 13:01:51.391903 (+ 230us) client_negotiation.cc:240] Negotiation successful
0808 13:01:51.392303 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":264,"threads_started":1}
I20260808 13:01:51.395607 21645 heartbeater.cc:347] Connected to a master server at 127.24.167.62:42273
I20260808 13:01:51.395987 21645 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:51.396863 21645 heartbeater.cc:511] Master 127.24.167.62:42273 requested a full tablet report, sending...
I20260808 13:01:51.398543 20960 ts_manager.cc:194] Registered new tserver with Master: 7b177b0128dc4d858a63a991a377cb1f (127.24.167.3:38811)
I20260808 13:01:51.399355 20960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:57689
I20260808 13:01:51.410104 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 13:01:51.427402 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.414610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.24.167.62:42273)
0808 13:01:51.414793 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:51.414798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:51.415199 (+ 401us) server_negotiation.cc:407] Connection header received
0808 13:01:51.415339 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:51.415343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:51.415407 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:51.415519 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:51.416961 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.417681 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.419202 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.419615 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.419711 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.422332 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:51.422337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:51.422340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:51.422388 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:51.424879 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:51.425597 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:51.425601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:51.425603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:51.425718 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:51.426098 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:51.426101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:51.426103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:51.426475 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:51.426680 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:51.427011 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 13:01:51.427171 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 13:01:51.431571 20960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 13:01:51.438720 20960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 13:01:51.464040 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.451093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:34957 (local address 127.0.0.1:52368)
0808 13:01:51.452050 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:51.452090 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:51.452273 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:51.453059 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:51.453067 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:51.453112 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:51.453596 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.453619 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.456454 (+ 2835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.456463 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.459690 (+ 3227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.459716 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.462536 (+ 2820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.462547 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.462883 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.462966 (+ 83us) client_negotiation.cc:769] Sending connection context
0808 13:01:51.463108 (+ 142us) client_negotiation.cc:240] Negotiation successful
0808 13:01:51.463395 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"spinlock_wait_cycles":7296,"thread_start_us":491,"threads_started":1}
I20260808 13:01:51.465607 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.451475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52368 (local address 127.24.167.1:34957)
0808 13:01:51.452149 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:51.452158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:51.452188 (+ 30us) server_negotiation.cc:407] Connection header received
0808 13:01:51.452515 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:51.452521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:51.452808 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:51.453835 (+ 1027us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:51.454151 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.456237 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.460056 (+ 3819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.462189 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.464712 (+ 2523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.464985 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:51.465064 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 13:01:51.465243 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":727,"threads_started":1}
I20260808 13:01:51.469543 21144 tablet_service.cc:1467] Processing CreateTablet for tablet ddd76b74ec634cb1b67a11f6ce5c4d12 (DEFAULT_TABLE table=test-table [id=f877fead21f443e1a1b0c6274be02dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:51.471258 21144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd76b74ec634cb1b67a11f6ce5c4d12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:51.472920 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.452778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55494 (local address 127.24.167.3:38811)
0808 13:01:51.465779 (+ 13001us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:51.465789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:51.465825 (+ 36us) server_negotiation.cc:407] Connection header received
0808 13:01:51.465933 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:51.465939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:51.466094 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:51.466299 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:51.466951 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.468489 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.470716 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.471713 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.471866 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.472101 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:51.472400 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 13:01:51.472587 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12830,"thread_start_us":12680,"threads_started":1}
I20260808 13:01:51.473029 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.453036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:35817 (local address 127.0.0.1:56772)
0808 13:01:51.464368 (+ 11332us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:51.464393 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:51.464518 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:51.465661 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:51.465669 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:51.465698 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:51.465867 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.465879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.468102 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.468108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.470179 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.470195 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.472083 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.472089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.472331 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.472365 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 13:01:51.472461 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 13:01:51.472640 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11200,"mutex_wait_us":88,"thread_start_us":11369,"threads_started":1}
I20260808 13:01:51.473280 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.452414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:38811 (local address 127.0.0.1:55494)
0808 13:01:51.452993 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:51.453013 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:51.453120 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:51.466283 (+ 13163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:51.466288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:51.466314 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:51.466740 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.466754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.468693 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.468699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.470542 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.470556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.471921 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.471928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.472199 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.472230 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 13:01:51.472342 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 13:01:51.472484 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":57,"spinlock_wait_cycles":10624,"thread_start_us":265,"threads_started":1}
I20260808 13:01:51.474290 21670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.453655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.24.167.2:35817)
0808 13:01:51.464964 (+ 11309us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:51.464974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:51.465007 (+ 33us) server_negotiation.cc:407] Connection header received
0808 13:01:51.465117 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:51.465128 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:51.465416 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:51.466069 (+ 653us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:51.466302 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.467923 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.470438 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.471887 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.473378 (+ 1491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.473727 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:51.473797 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 13:01:51.473967 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11119,"thread_start_us":11345,"threads_started":1}
I20260808 13:01:51.476753 21547 tablet_service.cc:1467] Processing CreateTablet for tablet ddd76b74ec634cb1b67a11f6ce5c4d12 (DEFAULT_TABLE table=test-table [id=f877fead21f443e1a1b0c6274be02dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:51.477679 21341 tablet_service.cc:1467] Processing CreateTablet for tablet ddd76b74ec634cb1b67a11f6ce5c4d12 (DEFAULT_TABLE table=test-table [id=f877fead21f443e1a1b0c6274be02dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:51.478271 21547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd76b74ec634cb1b67a11f6ce5c4d12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:51.479070 21341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd76b74ec634cb1b67a11f6ce5c4d12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:51.486613 21671 tablet_bootstrap.cc:492] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561: Bootstrap starting.
I20260808 13:01:51.491221 21673 tablet_bootstrap.cc:492] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f: Bootstrap starting.
I20260808 13:01:51.490518 21671 tablet_bootstrap.cc:654] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:51.492718 21674 tablet_bootstrap.cc:492] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306: Bootstrap starting.
I20260808 13:01:51.493778 21671 log.cc:824] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:51.495131 21673 tablet_bootstrap.cc:654] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:51.496107 21673 log.cc:824] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:51.496212 21674 tablet_bootstrap.cc:654] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:51.497323 21671 tablet_bootstrap.cc:492] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561: No bootstrap required, opened a new log
I20260808 13:01:51.497613 21671 ts_tablet_manager.cc:1397] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260808 13:01:51.497800 21674 log.cc:824] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:51.499006 21673 tablet_bootstrap.cc:492] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f: No bootstrap required, opened a new log
I20260808 13:01:51.499260 21673 ts_tablet_manager.cc:1397] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260808 13:01:51.500826 21674 tablet_bootstrap.cc:492] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306: No bootstrap required, opened a new log
I20260808 13:01:51.501088 21674 ts_tablet_manager.cc:1397] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260808 13:01:51.506986 21673 raft_consensus.cc:348] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59156f0786644e85a60f2bb82f6df561" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34957 } } peers { permanent_uuid: "7b177b0128dc4d858a63a991a377cb1f" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38811 } } peers { permanent_uuid: "6148332e10524fd686375a98516bb306" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35817 } }
I20260808 13:01:51.507401 21673 raft_consensus.cc:374] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:51.507503 21673 raft_consensus.cc:729] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b177b0128dc4d858a63a991a377cb1f, State: Initialized, Role: FOLLOWER
I20260808 13:01:51.508258 21673 consensus_queue.cc:260] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59156f0786644e85a60f2bb82f6df561" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34957 } } peers { permanent_uuid: "7b177b0128dc4d858a63a991a377cb1f" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38811 } } peers { permanent_uuid: "6148332e10524fd686375a98516bb306" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35817 } }
I20260808 13:01:51.509475 21645 heartbeater.cc:503] Master 127.24.167.62:42273 was elected leader, sending a full tablet report...
I20260808 13:01:51.510000 21673 ts_tablet_manager.cc:1428] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260808 13:01:51.510535 21671 raft_consensus.cc:348] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59156f0786644e85a60f2bb82f6df561" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34957 } } peers { permanent_uuid: "7b177b0128dc4d858a63a991a377cb1f" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38811 } } peers { permanent_uuid: "6148332e10524fd686375a98516bb306" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35817 } }
I20260808 13:01:51.510929 21671 raft_consensus.cc:374] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:51.511049 21671 raft_consensus.cc:729] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59156f0786644e85a60f2bb82f6df561, State: Initialized, Role: FOLLOWER
I20260808 13:01:51.511706 21671 consensus_queue.cc:260] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59156f0786644e85a60f2bb82f6df561" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34957 } } peers { permanent_uuid: "7b177b0128dc4d858a63a991a377cb1f" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38811 } } peers { permanent_uuid: "6148332e10524fd686375a98516bb306" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35817 } }
I20260808 13:01:51.511969 21674 raft_consensus.cc:348] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59156f0786644e85a60f2bb82f6df561" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34957 } } peers { permanent_uuid: "7b177b0128dc4d858a63a991a377cb1f" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38811 } } peers { permanent_uuid: "6148332e10524fd686375a98516bb306" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35817 } }
I20260808 13:01:51.512451 21674 raft_consensus.cc:374] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:51.512566 21674 raft_consensus.cc:729] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6148332e10524fd686375a98516bb306, State: Initialized, Role: FOLLOWER
I20260808 13:01:51.513155 21674 consensus_queue.cc:260] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59156f0786644e85a60f2bb82f6df561" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34957 } } peers { permanent_uuid: "7b177b0128dc4d858a63a991a377cb1f" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38811 } } peers { permanent_uuid: "6148332e10524fd686375a98516bb306" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35817 } }
I20260808 13:01:51.514492 21440 heartbeater.cc:503] Master 127.24.167.62:42273 was elected leader, sending a full tablet report...
I20260808 13:01:51.515125 21674 ts_tablet_manager.cc:1428] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306: Time spent starting tablet: real 0.014s user 0.008s sys 0.008s
I20260808 13:01:51.516782 21671 ts_tablet_manager.cc:1428] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
W20260808 13:01:51.561777 21441 tablet.cc:2369] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 13:01:51.623215 21647 tablet.cc:2369] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 13:01:51.758203 21241 tablet.cc:2369] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:01:51.891004 21682 raft_consensus.cc:482] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 13:01:51.891320 21682 raft_consensus.cc:504] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59156f0786644e85a60f2bb82f6df561" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34957 } } peers { permanent_uuid: "7b177b0128dc4d858a63a991a377cb1f" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38811 } } peers { permanent_uuid: "6148332e10524fd686375a98516bb306" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35817 } }
I20260808 13:01:51.893196 21682 leader_election.cc:288] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b177b0128dc4d858a63a991a377cb1f (127.24.167.3:38811), 6148332e10524fd686375a98516bb306 (127.24.167.2:35817)
I20260808 13:01:51.900164 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.893710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:43205 (local address 127.24.167.3:38811)
0808 13:01:51.893906 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:51.893911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:51.894254 (+ 343us) server_negotiation.cc:407] Connection header received
0808 13:01:51.894670 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:51.894676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:51.894759 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:51.894830 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:51.895563 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.896281 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.898277 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.899107 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.899290 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.899344 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:51.899741 (+ 397us) server_negotiation.cc:299] Negotiation successful
0808 13:01:51.899877 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 13:01:51.900264 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.893432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:38811 (local address 127.24.167.1:43205)
0808 13:01:51.894147 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:51.894169 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:51.894431 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:51.895018 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:51.895021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:51.895042 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:51.895409 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.895417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.896416 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.896424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.898120 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.898132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.899345 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.899351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.899529 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.899569 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 13:01:51.899678 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 13:01:51.899807 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":1,"spinlock_wait_cycles":18688,"thread_start_us":378,"threads_started":1}
I20260808 13:01:51.900578 21681 raft_consensus.cc:482] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 13:01:51.900866 21681 raft_consensus.cc:504] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59156f0786644e85a60f2bb82f6df561" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34957 } } peers { permanent_uuid: "7b177b0128dc4d858a63a991a377cb1f" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38811 } } peers { permanent_uuid: "6148332e10524fd686375a98516bb306" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35817 } }
I20260808 13:01:51.901469 21670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.894612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:49433 (local address 127.24.167.2:35817)
0808 13:01:51.894831 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:51.894837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:51.895099 (+ 262us) server_negotiation.cc:407] Connection header received
0808 13:01:51.895340 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:51.895344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:51.895403 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:51.895503 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:51.896046 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.896830 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.898754 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.899778 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.899921 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.899995 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:51.901074 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0808 13:01:51.901213 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 13:01:51.901474 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.894260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:35817 (local address 127.24.167.1:49433)
0808 13:01:51.894975 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:51.894995 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:51.895188 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:51.895517 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:51.895522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:51.895545 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:51.895885 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.895893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.897031 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.897035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.898600 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.898612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.900040 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.900047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.900871 (+ 824us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.900925 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 13:01:51.901026 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 13:01:51.901165 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":370,"threads_started":1}
I20260808 13:01:51.901389 21574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd76b74ec634cb1b67a11f6ce5c4d12" candidate_uuid: "59156f0786644e85a60f2bb82f6df561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b177b0128dc4d858a63a991a377cb1f" is_pre_election: true
I20260808 13:01:51.901816 21574 raft_consensus.cc:2454] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59156f0786644e85a60f2bb82f6df561 in term 0.
I20260808 13:01:51.902621 21070 leader_election.cc:302] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59156f0786644e85a60f2bb82f6df561, 7b177b0128dc4d858a63a991a377cb1f; no voters:
I20260808 13:01:51.902884 21681 leader_election.cc:288] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59156f0786644e85a60f2bb82f6df561 (127.24.167.1:34957), 6148332e10524fd686375a98516bb306 (127.24.167.2:35817)
I20260808 13:01:51.902810 21368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd76b74ec634cb1b67a11f6ce5c4d12" candidate_uuid: "59156f0786644e85a60f2bb82f6df561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6148332e10524fd686375a98516bb306" is_pre_election: true
I20260808 13:01:51.903067 21682 raft_consensus.cc:2781] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 13:01:51.903231 21682 raft_consensus.cc:482] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 13:01:51.903367 21682 raft_consensus.cc:3037] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:51.903406 21368 raft_consensus.cc:2454] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59156f0786644e85a60f2bb82f6df561 in term 0.
I20260808 13:01:51.907579 21682 raft_consensus.cc:504] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59156f0786644e85a60f2bb82f6df561" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34957 } } peers { permanent_uuid: "7b177b0128dc4d858a63a991a377cb1f" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38811 } } peers { permanent_uuid: "6148332e10524fd686375a98516bb306" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35817 } }
I20260808 13:01:51.908465 21682 leader_election.cc:288] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [CANDIDATE]: Term 1 election: Requested vote from peers 7b177b0128dc4d858a63a991a377cb1f (127.24.167.3:38811), 6148332e10524fd686375a98516bb306 (127.24.167.2:35817)
I20260808 13:01:51.909210 21574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd76b74ec634cb1b67a11f6ce5c4d12" candidate_uuid: "59156f0786644e85a60f2bb82f6df561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b177b0128dc4d858a63a991a377cb1f"
I20260808 13:01:51.909359 21368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd76b74ec634cb1b67a11f6ce5c4d12" candidate_uuid: "59156f0786644e85a60f2bb82f6df561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6148332e10524fd686375a98516bb306"
I20260808 13:01:51.909437 21574 raft_consensus.cc:3037] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:51.909582 21368 raft_consensus.cc:3037] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:51.910916 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.903448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:46581 (local address 127.24.167.1:34957)
0808 13:01:51.903640 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:51.903647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:51.903855 (+ 208us) server_negotiation.cc:407] Connection header received
0808 13:01:51.904109 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:51.904114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:51.904185 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:51.904250 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:51.904986 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.905724 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.908442 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.909647 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.909778 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.909860 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:51.910553 (+ 693us) server_negotiation.cc:299] Negotiation successful
0808 13:01:51.910685 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 13:01:51.911648 21686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.903038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:34957 (local address 127.24.167.3:46581)
0808 13:01:51.903754 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:51.903775 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:51.903955 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:51.904486 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:51.904490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:51.904518 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:51.904800 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.904813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.905860 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.905868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.908227 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.908242 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.910026 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.910034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.910308 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.910349 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 13:01:51.911123 (+ 774us) client_negotiation.cc:240] Negotiation successful
0808 13:01:51.911317 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":619,"threads_started":1}
I20260808 13:01:51.912495 21174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd76b74ec634cb1b67a11f6ce5c4d12" candidate_uuid: "7b177b0128dc4d858a63a991a377cb1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59156f0786644e85a60f2bb82f6df561" is_pre_election: true
I20260808 13:01:51.912968 21174 raft_consensus.cc:2379] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b177b0128dc4d858a63a991a377cb1f in current term 1: Already voted for candidate 59156f0786644e85a60f2bb82f6df561 in this term.
I20260808 13:01:51.912955 21574 raft_consensus.cc:2454] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59156f0786644e85a60f2bb82f6df561 in term 1.
I20260808 13:01:51.913350 21070 leader_election.cc:302] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59156f0786644e85a60f2bb82f6df561, 7b177b0128dc4d858a63a991a377cb1f; no voters:
I20260808 13:01:51.913604 21368 raft_consensus.cc:2454] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59156f0786644e85a60f2bb82f6df561 in term 1.
I20260808 13:01:51.913656 21682 raft_consensus.cc:2781] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:01:51.914644 21682 raft_consensus.cc:686] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [term 1 LEADER]: Becoming Leader. State: Replica: 59156f0786644e85a60f2bb82f6df561, State: Running, Role: LEADER
I20260808 13:01:51.915145 21682 consensus_queue.cc:237] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59156f0786644e85a60f2bb82f6df561" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34957 } } peers { permanent_uuid: "7b177b0128dc4d858a63a991a377cb1f" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38811 } } peers { permanent_uuid: "6148332e10524fd686375a98516bb306" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35817 } }
I20260808 13:01:51.917896 21670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.904452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:53709 (local address 127.24.167.2:35817)
0808 13:01:51.904609 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:51.904614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:51.909236 (+ 4622us) server_negotiation.cc:407] Connection header received
0808 13:01:51.909525 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:51.909530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:51.909610 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:51.909674 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:51.910658 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.911840 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.915731 (+ 3891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.916956 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.917119 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.917193 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:51.917591 (+ 398us) server_negotiation.cc:299] Negotiation successful
0808 13:01:51.917720 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 13:01:51.918761 21687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:51.904047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:35817 (local address 127.24.167.3:53709)
0808 13:01:51.909129 (+ 5082us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:51.909147 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:51.909339 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:51.909882 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:51.909887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:51.909918 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:51.910459 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.910472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.912028 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.912033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.915574 (+ 3541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:51.915587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:51.917148 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:51.917153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:51.917312 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:51.917361 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 13:01:51.917524 (+ 163us) client_negotiation.cc:240] Negotiation successful
0808 13:01:51.918537 (+ 1013us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4945,"thread_start_us":4736,"threads_started":1}
I20260808 13:01:51.919394 21368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd76b74ec634cb1b67a11f6ce5c4d12" candidate_uuid: "7b177b0128dc4d858a63a991a377cb1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6148332e10524fd686375a98516bb306" is_pre_election: true
I20260808 13:01:51.919720 21368 raft_consensus.cc:2379] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b177b0128dc4d858a63a991a377cb1f in current term 1: Already voted for candidate 59156f0786644e85a60f2bb82f6df561 in this term.
I20260808 13:01:51.920344 21476 leader_election.cc:302] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b177b0128dc4d858a63a991a377cb1f; no voters: 59156f0786644e85a60f2bb82f6df561, 6148332e10524fd686375a98516bb306
I20260808 13:01:51.920768 21681 raft_consensus.cc:2726] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 13:01:51.920718 20955 catalog_manager.cc:5495] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59156f0786644e85a60f2bb82f6df561 (127.24.167.1). New cstate: current_term: 1 leader_uuid: "59156f0786644e85a60f2bb82f6df561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59156f0786644e85a60f2bb82f6df561" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 34957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b177b0128dc4d858a63a991a377cb1f" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6148332e10524fd686375a98516bb306" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35817 } health_report { overall_health: UNKNOWN } } }
I20260808 13:01:52.035605 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:52.029443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.24.167.1:34957)
0808 13:01:52.029623 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:52.029629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:52.030041 (+ 412us) server_negotiation.cc:407] Connection header received
0808 13:01:52.030179 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:52.030183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:52.030312 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:52.030531 (+ 219us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:01:52.031017 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:52.031875 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:52.033330 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:52.033754 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:52.033863 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:52.034487 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:01:52.035048 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:01:52.035260 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:52.035315 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 13:01:52.035366 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 13:01:52.044133 21368 raft_consensus.cc:1261] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306 [term 1 FOLLOWER]: Refusing update from remote peer 59156f0786644e85a60f2bb82f6df561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:01:52.044169 21574 raft_consensus.cc:1261] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f [term 1 FOLLOWER]: Refusing update from remote peer 59156f0786644e85a60f2bb82f6df561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:01:52.045023 21682 consensus_queue.cc:1035] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b177b0128dc4d858a63a991a377cb1f" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:01:52.045400 21690 consensus_queue.cc:1035] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6148332e10524fd686375a98516bb306" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 35817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:01:52.061379 21698 mvcc.cc:205] Tried to move back new op lower bound from 7316251082922475520 to 7316251082408808448. Current Snapshot: MvccSnapshot[applied={T|T < 7316251082922475520}]
I20260808 13:01:52.066092 21697 mvcc.cc:205] Tried to move back new op lower bound from 7316251082922475520 to 7316251082408808448. Current Snapshot: MvccSnapshot[applied={T|T < 7316251082922475520 or (T in {7316251082922475520})}]
I20260808 13:01:52.073112 20955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:01:52.073388 20955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:01:52.076907 20955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4393e033aa9a4b50937d04485e3f873a: Sending DeleteTablet for 3 replicas of tablet ddd76b74ec634cb1b67a11f6ce5c4d12
I20260808 13:01:52.078030 21144 tablet_service.cc:1514] Processing DeleteTablet for tablet ddd76b74ec634cb1b67a11f6ce5c4d12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:52 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:52368
I20260808 13:01:52.078220 21341 tablet_service.cc:1514] Processing DeleteTablet for tablet ddd76b74ec634cb1b67a11f6ce5c4d12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:52 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:56772
I20260808 13:01:52.078220 21547 tablet_service.cc:1514] Processing DeleteTablet for tablet ddd76b74ec634cb1b67a11f6ce5c4d12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:52 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:55494
I20260808 13:01:52.080446 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 21042
I20260808 13:01:52.089557 21715 tablet_replica.cc:330] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561: stopping tablet replica
I20260808 13:01:52.090188 21715 raft_consensus.cc:2229] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [term 1 LEADER]: Raft consensus shutting down.
I20260808 13:01:52.091051 21715 raft_consensus.cc:2258] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:52.093497 21715 ts_tablet_manager.cc:1905] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:52.101364 21715 ts_tablet_manager.cc:1918] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:52.101624 21715 log.cc:1196] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/ddd76b74ec634cb1b67a11f6ce5c4d12
I20260808 13:01:52.102341 21715 ts_tablet_manager.cc:1939] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 59156f0786644e85a60f2bb82f6df561: Deleting consensus metadata
I20260808 13:01:52.104465 20938 catalog_manager.cc:4841] TS 59156f0786644e85a60f2bb82f6df561 (127.24.167.1:34957): tablet ddd76b74ec634cb1b67a11f6ce5c4d12 (table test-table [id=f877fead21f443e1a1b0c6274be02dc5]) successfully deleted
I20260808 13:01:52.105762 21716 tablet_replica.cc:330] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306: stopping tablet replica
I20260808 13:01:52.106307 21716 raft_consensus.cc:2229] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:01:52.106957 21716 raft_consensus.cc:2258] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:52.107829 21717 tablet_replica.cc:330] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f: stopping tablet replica
I20260808 13:01:52.108331 21717 raft_consensus.cc:2229] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:01:52.108867 21717 raft_consensus.cc:2258] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:52.109143 21716 ts_tablet_manager.cc:1905] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:52.110841 21717 ts_tablet_manager.cc:1905] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 13:01:52 dist-test-slave-tk9z krb5kdc[20896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 13:01:52.117220 21716 ts_tablet_manager.cc:1918] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:52.117424 21716 log.cc:1196] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/ddd76b74ec634cb1b67a11f6ce5c4d12
I20260808 13:01:52.117976 21716 ts_tablet_manager.cc:1939] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 6148332e10524fd686375a98516bb306: Deleting consensus metadata
I20260808 13:01:52.118541 21717 ts_tablet_manager.cc:1918] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:52.118768 21717 log.cc:1196] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/ddd76b74ec634cb1b67a11f6ce5c4d12
I20260808 13:01:52.119346 21717 ts_tablet_manager.cc:1939] T ddd76b74ec634cb1b67a11f6ce5c4d12 P 7b177b0128dc4d858a63a991a377cb1f: Deleting consensus metadata
I20260808 13:01:52.120483 20938 catalog_manager.cc:4841] TS 6148332e10524fd686375a98516bb306 (127.24.167.2:35817): tablet ddd76b74ec634cb1b67a11f6ce5c4d12 (table test-table [id=f877fead21f443e1a1b0c6274be02dc5]) successfully deleted
I20260808 13:01:52.121302 20938 catalog_manager.cc:4841] TS 7b177b0128dc4d858a63a991a377cb1f (127.24.167.3:38811): tablet ddd76b74ec634cb1b67a11f6ce5c4d12 (table test-table [id=f877fead21f443e1a1b0c6274be02dc5]) successfully deleted
I20260808 13:01:52.125244 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:52.082172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52396 (local address 127.24.167.1:34957)
0808 13:01:52.082435 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:52.082444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:52.105009 (+ 22565us) server_negotiation.cc:407] Connection header received
0808 13:01:52.105215 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:52.105220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:52.105361 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:52.105526 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:52.107937 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:52.109088 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:52.111123 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:52.111821 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:52.111941 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:52.118105 (+ 6164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:52.118128 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:52.118151 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:52.118207 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:52.122158 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:52.122995 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:52.123003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:52.123005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:52.123170 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:52.123584 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:52.123588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:52.123590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:52.123962 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:52.124311 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:52.124639 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 13:01:52.124904 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 13:01:52.125943 21232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:52.226516 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 21042
I20260808 13:01:52.253261 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 21250
Aug 08 13:01:52 dist-test-slave-tk9z krb5kdc[20896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 13:01:52.268863 21670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:52.254072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56788 (local address 127.24.167.2:35817)
0808 13:01:52.254299 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:52.254307 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:52.254400 (+ 93us) server_negotiation.cc:407] Connection header received
0808 13:01:52.254640 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:52.254644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:52.254717 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:52.254860 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:52.256307 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:52.257046 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:52.258403 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:52.258929 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:52.259082 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:52.263515 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:52.263530 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:52.263548 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:52.263582 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:52.266532 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:52.267158 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:52.267165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:52.267168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:52.267284 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:52.267609 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:52.267612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:52.267614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:52.267964 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:52.268198 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:52.268427 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 13:01:52.268597 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 13:01:52.269593 21432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:52.360376 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 21250
I20260808 13:01:52.384788 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 21447
Aug 08 13:01:52 dist-test-slave-tk9z krb5kdc[20896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 13:01:52.402717 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:52.385655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.24.167.3:38811)
0808 13:01:52.385857 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:52.385861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:52.385934 (+ 73us) server_negotiation.cc:407] Connection header received
0808 13:01:52.386103 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:52.386106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:52.386197 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:52.386286 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:52.388014 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:52.389265 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:52.390785 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:52.391178 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:52.391271 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:52.395717 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:52.395729 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:52.395745 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:52.395786 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:52.399998 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:52.400757 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:52.400764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:52.400767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:52.400891 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:52.401251 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:52.401255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:52.401257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:52.401636 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:52.401863 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:52.402134 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 13:01:52.402307 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 13:01:52.403440 21637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:52.494225 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 21447
I20260808 13:01:52.520138 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 20913
I20260808 13:01:52.521448 21002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:52.635433 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 20913
2026-08-08T13:01:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5160 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T13:01:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:01:52Z Disabled control of system clock
I20260808 13:01:52.736845 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:44085
--webserver_interface=127.24.167.62
--webserver_port=0
--builtin_ntp_servers=127.24.167.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:44085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 13:01:53.095192 21729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:53.095496 21729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:53.095580 21729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:53.102909 21729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:01:53.103042 21729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:53.103081 21729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:53.103140 21729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:01:53.103185 21729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:01:53.112792 21729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:44085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:44085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:53.114182 21729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:53.116086 21729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:53.128499 21741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:53.128533 21737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:53.128747 21743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:53.129374 21729 server_base.cc:1034] running on GCE node
I20260808 13:01:53.130340 21729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:53.132067 21729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:53.133388 21729 hybrid_clock.cc:648] HybridClock initialized: now 1786194113133386 us; error 110 us; skew 500 ppm
I20260808 13:01:53.133823 21729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 13:01:53.136415 21729 webserver.cc:460] Webserver started at http://127.24.167.62:38905/ using document root <none> and password file <none>
I20260808 13:01:53.137157 21729 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:53.137312 21729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:53.137676 21729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:53.140887 21729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "d4e30afa846b41d683b5eb25a298f4b5"
format_stamp: "Formatted at 2026-08-08 13:01:53 on dist-test-slave-tk9z"
server_key: "0f55c78919894c614a020fb3f0668299"
server_key_iv: "733a14e6b68522032d8e606b6b6418c8"
server_key_version: "encryptionkey@0"
I20260808 13:01:53.141775 21729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "d4e30afa846b41d683b5eb25a298f4b5"
format_stamp: "Formatted at 2026-08-08 13:01:53 on dist-test-slave-tk9z"
server_key: "0f55c78919894c614a020fb3f0668299"
server_key_iv: "733a14e6b68522032d8e606b6b6418c8"
server_key_version: "encryptionkey@0"
I20260808 13:01:53.147184 21729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260808 13:01:53.150868 21753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:53.152364 21729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 13:01:53.152592 21729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "d4e30afa846b41d683b5eb25a298f4b5"
format_stamp: "Formatted at 2026-08-08 13:01:53 on dist-test-slave-tk9z"
server_key: "0f55c78919894c614a020fb3f0668299"
server_key_iv: "733a14e6b68522032d8e606b6b6418c8"
server_key_version: "encryptionkey@0"
I20260808 13:01:53.152838 21729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:53.183751 21729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:53.184672 21729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:53.184990 21729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:53.204677 21729 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:44085
I20260808 13:01:53.204691 21819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:44085 every 8 connection(s)
I20260808 13:01:53.206146 21729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:01:53.209937 21820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:53.215749 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 21729
I20260808 13:01:53.215931 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:01:53.216423 25244 external_mini_cluster.cc:1428] Setting key 257feda333a3664b60282599da4ca8b3
I20260808 13:01:53.222182 21820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5: Bootstrap starting.
I20260808 13:01:53.226328 21820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:53.227507 21820 log.cc:824] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:53.231004 21820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5: No bootstrap required, opened a new log
I20260808 13:01:53.232928 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:53.218528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.24.167.62:44085)
0808 13:01:53.219644 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:53.219660 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:53.219712 (+ 52us) server_negotiation.cc:407] Connection header received
0808 13:01:53.222193 (+ 2481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:53.222226 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:53.222868 (+ 642us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:53.223314 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:53.223891 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:53.225943 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:53.227498 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:53.228494 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:53.228630 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:53.229955 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:53.229968 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:53.229990 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:53.230745 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:53.230849 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:53.230875 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:53.231032 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:53.231263 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 13:01:53.231620 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":325,"threads_started":1}
I20260808 13:01:53.239959 21820 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e30afa846b41d683b5eb25a298f4b5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44085 } }
I20260808 13:01:53.240310 21820 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:53.240398 21820 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e30afa846b41d683b5eb25a298f4b5, State: Initialized, Role: FOLLOWER
I20260808 13:01:53.240886 21820 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e30afa846b41d683b5eb25a298f4b5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44085 } }
I20260808 13:01:53.241104 21820 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:01:53.241200 21820 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:01:53.241330 21820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:53.243590 21820 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e30afa846b41d683b5eb25a298f4b5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44085 } }
I20260808 13:01:53.244098 21820 leader_election.cc:302] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4e30afa846b41d683b5eb25a298f4b5; no voters:
I20260808 13:01:53.244692 21820 leader_election.cc:288] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:01:53.244894 21832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:01:53.246027 21832 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [term 1 LEADER]: Becoming Leader. State: Replica: d4e30afa846b41d683b5eb25a298f4b5, State: Running, Role: LEADER
I20260808 13:01:53.246601 21832 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e30afa846b41d683b5eb25a298f4b5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44085 } }
I20260808 13:01:53.247110 21820 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:01:53.249683 21834 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4e30afa846b41d683b5eb25a298f4b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e30afa846b41d683b5eb25a298f4b5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44085 } } }
I20260808 13:01:53.249752 21835 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4e30afa846b41d683b5eb25a298f4b5. Latest consensus state: current_term: 1 leader_uuid: "d4e30afa846b41d683b5eb25a298f4b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e30afa846b41d683b5eb25a298f4b5" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44085 } } }
I20260808 13:01:53.250144 21834 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [sys.catalog]: This master's current role is: LEADER
I20260808 13:01:53.250151 21835 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5 [sys.catalog]: This master's current role is: LEADER
I20260808 13:01:53.252121 21841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:01:53.257685 21841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:01:53.267534 21841 catalog_manager.cc:1347] Generated new cluster ID: b3718fb0048d4f6b88d401acc813587d
I20260808 13:01:53.267670 21841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:01:53.281551 21841 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:01:53.283030 21841 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:01:53.294749 21841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5: Generated new TSK 0
I20260808 13:01:53.295707 21841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:01:53.308955 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44085
--builtin_ntp_servers=127.24.167.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 13:01:53.669624 21859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:53.669920 21859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:53.670008 21859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:53.677301 21859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:53.677410 21859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:53.677517 21859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 13:01:53.686689 21859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:53.687963 21859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:53.689872 21859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:53.702848 21870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:53.702951 21868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:53.702878 21867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:53.703790 21859 server_base.cc:1034] running on GCE node
I20260808 13:01:53.704391 21859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:53.705371 21859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:53.706597 21859 hybrid_clock.cc:648] HybridClock initialized: now 1786194113706539 us; error 52 us; skew 500 ppm
I20260808 13:01:53.706918 21859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 13:01:53.709208 21859 webserver.cc:460] Webserver started at http://127.24.167.1:40351/ using document root <none> and password file <none>
I20260808 13:01:53.709882 21859 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:53.710021 21859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:53.710351 21859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:53.713204 21859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "28d923934ea54667a508d559a6388efe"
format_stamp: "Formatted at 2026-08-08 13:01:53 on dist-test-slave-tk9z"
server_key: "b1f5c384ad60b13d3fe3f1552b8fac2a"
server_key_iv: "fa1f9a019ce700831b71616fc27bafd2"
server_key_version: "encryptionkey@0"
I20260808 13:01:53.713981 21859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "28d923934ea54667a508d559a6388efe"
format_stamp: "Formatted at 2026-08-08 13:01:53 on dist-test-slave-tk9z"
server_key: "b1f5c384ad60b13d3fe3f1552b8fac2a"
server_key_iv: "fa1f9a019ce700831b71616fc27bafd2"
server_key_version: "encryptionkey@0"
I20260808 13:01:53.719449 21859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 13:01:53.722992 21880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:53.724432 21859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 13:01:53.724613 21859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "28d923934ea54667a508d559a6388efe"
format_stamp: "Formatted at 2026-08-08 13:01:53 on dist-test-slave-tk9z"
server_key: "b1f5c384ad60b13d3fe3f1552b8fac2a"
server_key_iv: "fa1f9a019ce700831b71616fc27bafd2"
server_key_version: "encryptionkey@0"
I20260808 13:01:53.724845 21859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:53.764675 21859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:53.765570 21859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:53.765905 21859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:53.767009 21859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:53.768602 21859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:53.768731 21859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:53.768855 21859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:53.768946 21859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:53.803443 21859 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:35587
I20260808 13:01:53.803514 22052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:35587 every 8 connection(s)
I20260808 13:01:53.804966 21859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 13:01:53.810814 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 21859
I20260808 13:01:53.811024 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 13:01:53.811565 25244 external_mini_cluster.cc:1428] Setting key 9bdfe9ae874a9b1715c9db7f01a58600
I20260808 13:01:53.814970 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44085
--builtin_ntp_servers=127.24.167.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 13:01:53.819442 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:53.807567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:57531 (local address 127.24.167.62:44085)
0808 13:01:53.808051 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:53.808055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:53.809047 (+ 992us) server_negotiation.cc:407] Connection header received
0808 13:01:53.811202 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:53.811207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:53.811279 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:53.811375 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:53.812725 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:53.813530 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:53.816124 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:53.816765 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:53.816895 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:53.818661 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:53.818666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:53.818669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:53.818687 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:53.818733 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:53.818745 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:53.818826 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:53.819103 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 13:01:53.819187 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":265,"threads_started":1}
I20260808 13:01:53.820971 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:53.808007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:44085 (local address 127.24.167.1:57531)
0808 13:01:53.808888 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:53.808922 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:53.810837 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:53.811667 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:53.811679 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:53.811804 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:53.812532 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:53.812550 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:53.813718 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:53.813722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:53.815879 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:53.815898 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:53.817025 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:53.817035 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:53.817401 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:53.818213 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 13:01:53.818219 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:53.818280 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 13:01:53.818425 (+ 145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:53.818853 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:53.818867 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:53.818878 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 13:01:53.819062 (+ 184us) client_negotiation.cc:240] Negotiation successful
0808 13:01:53.819517 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":271,"threads_started":1}
I20260808 13:01:53.822629 22054 heartbeater.cc:347] Connected to a master server at 127.24.167.62:44085
I20260808 13:01:53.822979 22054 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:53.823984 22054 heartbeater.cc:511] Master 127.24.167.62:44085 requested a full tablet report, sending...
I20260808 13:01:53.826333 21776 ts_manager.cc:194] Registered new tserver with Master: 28d923934ea54667a508d559a6388efe (127.24.167.1:35587)
I20260808 13:01:53.828315 21776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.167.1:57531
W20260808 13:01:54.163806 22058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:54.164098 22058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:54.164242 22058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:54.171536 22058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:54.171634 22058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:54.171808 22058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 13:01:54.180923 22058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:54.182243 22058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:54.184037 22058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:54.197238 22066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:54.197273 22070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:54.197268 22067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:54.198170 22058 server_base.cc:1034] running on GCE node
I20260808 13:01:54.198868 22058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:54.199774 22058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:54.201010 22058 hybrid_clock.cc:648] HybridClock initialized: now 1786194114200977 us; error 57 us; skew 500 ppm
I20260808 13:01:54.201342 22058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 13:01:54.203585 22058 webserver.cc:460] Webserver started at http://127.24.167.2:43211/ using document root <none> and password file <none>
I20260808 13:01:54.204502 22058 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:54.204638 22058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:54.204998 22058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:54.208732 22058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "d73251e095b8425e8ca13eb5d96aedf1"
format_stamp: "Formatted at 2026-08-08 13:01:54 on dist-test-slave-tk9z"
server_key: "03e76e8071ef34df31f246ab8c939dc7"
server_key_iv: "49d81d624aefada3fa2ca2ca544d87fa"
server_key_version: "encryptionkey@0"
I20260808 13:01:54.209698 22058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "d73251e095b8425e8ca13eb5d96aedf1"
format_stamp: "Formatted at 2026-08-08 13:01:54 on dist-test-slave-tk9z"
server_key: "03e76e8071ef34df31f246ab8c939dc7"
server_key_iv: "49d81d624aefada3fa2ca2ca544d87fa"
server_key_version: "encryptionkey@0"
I20260808 13:01:54.215832 22058 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260808 13:01:54.220083 22079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:54.221626 22058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 13:01:54.221787 22058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "d73251e095b8425e8ca13eb5d96aedf1"
format_stamp: "Formatted at 2026-08-08 13:01:54 on dist-test-slave-tk9z"
server_key: "03e76e8071ef34df31f246ab8c939dc7"
server_key_iv: "49d81d624aefada3fa2ca2ca544d87fa"
server_key_version: "encryptionkey@0"
I20260808 13:01:54.222005 22058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:54.240239 22058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:54.241029 22058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:54.241294 22058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:54.242235 22058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:54.243714 22058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:54.243806 22058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:54.243888 22058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:54.243943 22058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:54.277530 22058 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:42759
I20260808 13:01:54.277581 22236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:42759 every 8 connection(s)
I20260808 13:01:54.278975 22058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 13:01:54.288908 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 22058
I20260808 13:01:54.289099 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 13:01:54.289688 25244 external_mini_cluster.cc:1428] Setting key 29cd44aa5bc51ef51bd86c81a6b9b7ed
I20260808 13:01:54.291857 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.281601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:45813 (local address 127.24.167.62:44085)
0808 13:01:54.281786 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:54.281790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:54.283189 (+ 1399us) server_negotiation.cc:407] Connection header received
0808 13:01:54.284987 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:54.284993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:54.285082 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:54.285208 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:54.286280 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.287074 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.288700 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.289272 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.289375 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.291007 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:54.291011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:54.291014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:54.291034 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:54.291070 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:54.291082 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:54.291179 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:54.291414 (+ 235us) server_negotiation.cc:299] Negotiation successful
0808 13:01:54.291518 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 13:01:54.292742 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44085
--builtin_ntp_servers=127.24.167.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 13:01:54.292899 22239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.282116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:44085 (local address 127.24.167.2:45813)
0808 13:01:54.283033 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:54.283074 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:54.284710 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:54.285440 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:54.285453 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:54.285534 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:54.286108 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:54.286125 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.287244 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.287249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:54.288548 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:54.288565 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.289515 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.289524 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:54.289825 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.290642 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 13:01:54.290645 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:54.290685 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 13:01:54.290767 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:54.291194 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:54.291209 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:54.291221 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 13:01:54.291398 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 13:01:54.291808 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":331,"threads_started":1}
I20260808 13:01:54.294327 22237 heartbeater.cc:347] Connected to a master server at 127.24.167.62:44085
I20260808 13:01:54.294672 22237 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:54.295537 22237 heartbeater.cc:511] Master 127.24.167.62:44085 requested a full tablet report, sending...
I20260808 13:01:54.297103 21776 ts_manager.cc:194] Registered new tserver with Master: d73251e095b8425e8ca13eb5d96aedf1 (127.24.167.2:42759)
I20260808 13:01:54.298333 21776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.167.2:45813
W20260808 13:01:54.639305 22240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:54.639662 22240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:54.639832 22240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:54.647082 22240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:54.647233 22240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:54.647425 22240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 13:01:54.656963 22240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:44085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:54.658435 22240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:54.660223 22240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:54.673403 22248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:54.673358 22247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:54.673461 22250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:54.674185 22240 server_base.cc:1034] running on GCE node
I20260808 13:01:54.674927 22240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:54.675942 22240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:54.677178 22240 hybrid_clock.cc:648] HybridClock initialized: now 1786194114677124 us; error 60 us; skew 500 ppm
I20260808 13:01:54.677531 22240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 13:01:54.679840 22240 webserver.cc:460] Webserver started at http://127.24.167.3:33291/ using document root <none> and password file <none>
I20260808 13:01:54.680545 22240 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:54.680681 22240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:54.681016 22240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:54.683894 22240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "b407bd25892c4f0f948b48424ff93307"
format_stamp: "Formatted at 2026-08-08 13:01:54 on dist-test-slave-tk9z"
server_key: "cdc44819327a31e77a640310e32555f3"
server_key_iv: "0c90f1461d4ed408cdc6f71ef5c77dac"
server_key_version: "encryptionkey@0"
I20260808 13:01:54.684687 22240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "b407bd25892c4f0f948b48424ff93307"
format_stamp: "Formatted at 2026-08-08 13:01:54 on dist-test-slave-tk9z"
server_key: "cdc44819327a31e77a640310e32555f3"
server_key_iv: "0c90f1461d4ed408cdc6f71ef5c77dac"
server_key_version: "encryptionkey@0"
I20260808 13:01:54.689591 22240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 13:01:54.693240 22257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:54.694602 22240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 13:01:54.694769 22240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "b407bd25892c4f0f948b48424ff93307"
format_stamp: "Formatted at 2026-08-08 13:01:54 on dist-test-slave-tk9z"
server_key: "cdc44819327a31e77a640310e32555f3"
server_key_iv: "0c90f1461d4ed408cdc6f71ef5c77dac"
server_key_version: "encryptionkey@0"
I20260808 13:01:54.694989 22240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:54.715890 22240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:54.716655 22240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:54.716964 22240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:54.717908 22240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:54.719540 22240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:54.719661 22240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:54.719794 22240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:54.719882 22240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:54.754827 22240 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:38117
I20260808 13:01:54.754930 22422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:38117 every 8 connection(s)
I20260808 13:01:54.756152 22240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 13:01:54.763861 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 22240
I20260808 13:01:54.764047 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 13:01:54.764545 25244 external_mini_cluster.cc:1428] Setting key e7ee623318501bcd504e293ac90f7fd9
I20260808 13:01:54.769199 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.758965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:45147 (local address 127.24.167.62:44085)
0808 13:01:54.759174 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:54.759179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:54.760500 (+ 1321us) server_negotiation.cc:407] Connection header received
0808 13:01:54.762254 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:54.762259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:54.762330 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:54.762530 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:54.763513 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.764407 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.766403 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.766848 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.766991 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.768352 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:54.768356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:54.768359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:54.768376 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:54.768410 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:54.768426 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:54.768523 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:54.768744 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 13:01:54.768842 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 13:01:54.770185 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.759480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:44085 (local address 127.24.167.3:45147)
0808 13:01:54.760363 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:54.760410 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:54.761974 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:54.762694 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:54.762701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:54.762785 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:54.763332 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:54.763348 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.764566 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.764574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:54.766231 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:54.766243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.767076 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.767085 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:54.767330 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.768038 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 13:01:54.768041 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:54.768080 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 13:01:54.768159 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:54.768525 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:54.768538 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:54.768548 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 13:01:54.768732 (+ 184us) client_negotiation.cc:240] Negotiation successful
0808 13:01:54.769116 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":284,"threads_started":1}
I20260808 13:01:54.771584 22423 heartbeater.cc:347] Connected to a master server at 127.24.167.62:44085
I20260808 13:01:54.771900 22423 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:54.772692 22423 heartbeater.cc:511] Master 127.24.167.62:44085 requested a full tablet report, sending...
I20260808 13:01:54.774197 21776 ts_manager.cc:194] Registered new tserver with Master: b407bd25892c4f0f948b48424ff93307 (127.24.167.3:38117)
I20260808 13:01:54.775063 21776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.167.3:45147
I20260808 13:01:54.779949 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 13:01:54.790614 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.784158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37174 (local address 127.24.167.62:44085)
0808 13:01:54.784376 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:54.784384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:54.784755 (+ 371us) server_negotiation.cc:407] Connection header received
0808 13:01:54.784970 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:54.784974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:54.785039 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:54.785145 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:54.785698 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.786543 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.787954 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.788395 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.788531 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.789763 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:54.789775 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:54.789779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:54.789799 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:54.789847 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:54.789867 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:54.790041 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:54.790242 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 13:01:54.790317 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 13:01:54.794664 21776 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 13:01:54.801855 21776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 13:01:54.826980 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.815518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44282 (local address 127.24.167.3:38117)
0808 13:01:54.817799 (+ 2281us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:54.817813 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:54.817856 (+ 43us) server_negotiation.cc:407] Connection header received
0808 13:01:54.817977 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:54.817988 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:54.818614 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:54.818846 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:54.820349 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.821958 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.824792 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.825709 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.825890 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.826175 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:54.826394 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 13:01:54.826580 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2059,"thread_start_us":1858,"threads_started":1}
I20260808 13:01:54.828212 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.815148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:38117 (local address 127.0.0.1:44282)
0808 13:01:54.816816 (+ 1668us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:54.816863 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:54.817033 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:54.819225 (+ 2192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:54.819237 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:54.819287 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:54.819985 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:54.820006 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.822113 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.822121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:54.824589 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:54.824602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.825935 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.825943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:54.826129 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.826176 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 13:01:54.826305 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 13:01:54.827768 (+ 1463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1434,"mutex_wait_us":49,"thread_start_us":2614,"threads_started":1}
I20260808 13:01:54.829094 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.818528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.24.167.2:42759)
0808 13:01:54.820807 (+ 2279us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:54.820818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:54.820847 (+ 29us) server_negotiation.cc:407] Connection header received
0808 13:01:54.820949 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:54.820955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:54.821172 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:54.821378 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:54.821979 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.823447 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.825829 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.827345 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.827463 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.827698 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:54.828590 (+ 892us) server_negotiation.cc:299] Negotiation successful
0808 13:01:54.828752 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2119,"thread_start_us":2013,"threads_started":1}
I20260808 13:01:54.829567 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.816227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50096 (local address 127.24.167.1:35587)
0808 13:01:54.821151 (+ 4924us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:54.821159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:54.821187 (+ 28us) server_negotiation.cc:407] Connection header received
0808 13:01:54.821285 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:54.821294 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:54.821515 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:54.821728 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:54.822522 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.824135 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.826646 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.828204 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.828360 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.828637 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:54.829021 (+ 384us) server_negotiation.cc:299] Negotiation successful
0808 13:01:54.829208 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4755,"thread_start_us":4660,"threads_started":1}
I20260808 13:01:54.829650 22439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.815685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:35587 (local address 127.0.0.1:50096)
0808 13:01:54.818068 (+ 2383us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:54.818090 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:54.818212 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:54.821728 (+ 3516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:54.821733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:54.821757 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:54.822273 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:54.822290 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.824287 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.824293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:54.826427 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:54.826441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.828559 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.828565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:54.828763 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.828808 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 13:01:54.828969 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 13:01:54.829138 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2232,"thread_start_us":758,"threads_started":1}
I20260808 13:01:54.829993 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.818818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:42759 (local address 127.0.0.1:55970)
0808 13:01:54.820134 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:54.820156 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:54.820291 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:54.821383 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:54.821388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:54.821416 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:54.821741 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:54.821752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.823580 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.823587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:54.825612 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:54.825627 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.827958 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.827966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:54.828205 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.828242 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 13:01:54.828344 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 13:01:54.829492 (+ 1148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"thread_start_us":991,"threads_started":1}
I20260808 13:01:54.832911 22321 tablet_service.cc:1467] Processing CreateTablet for tablet 36eed835df004f76bd10a5e4cd1b814b (DEFAULT_TABLE table=test-table [id=335b9820658e4b19bf1c6a2382738e34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:54.833236 21951 tablet_service.cc:1467] Processing CreateTablet for tablet 36eed835df004f76bd10a5e4cd1b814b (DEFAULT_TABLE table=test-table [id=335b9820658e4b19bf1c6a2382738e34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:54.833496 22144 tablet_service.cc:1467] Processing CreateTablet for tablet 36eed835df004f76bd10a5e4cd1b814b (DEFAULT_TABLE table=test-table [id=335b9820658e4b19bf1c6a2382738e34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:54.834527 21951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36eed835df004f76bd10a5e4cd1b814b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:54.834559 22321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36eed835df004f76bd10a5e4cd1b814b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:54.834803 22144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36eed835df004f76bd10a5e4cd1b814b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:54.838935 22054 heartbeater.cc:503] Master 127.24.167.62:44085 was elected leader, sending a full tablet report...
I20260808 13:01:54.846836 22446 tablet_bootstrap.cc:492] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe: Bootstrap starting.
I20260808 13:01:54.849148 22449 tablet_bootstrap.cc:492] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307: Bootstrap starting.
I20260808 13:01:54.849948 22447 tablet_bootstrap.cc:492] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1: Bootstrap starting.
I20260808 13:01:54.851078 22446 tablet_bootstrap.cc:654] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:54.852434 22446 log.cc:824] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:54.853166 22449 tablet_bootstrap.cc:654] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:54.853631 22447 tablet_bootstrap.cc:654] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:54.854671 22449 log.cc:824] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:54.854919 22447 log.cc:824] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:54.856627 22446 tablet_bootstrap.cc:492] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe: No bootstrap required, opened a new log
I20260808 13:01:54.856947 22446 ts_tablet_manager.cc:1397] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260808 13:01:54.857879 22447 tablet_bootstrap.cc:492] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1: No bootstrap required, opened a new log
I20260808 13:01:54.858003 22449 tablet_bootstrap.cc:492] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307: No bootstrap required, opened a new log
I20260808 13:01:54.858206 22447 ts_tablet_manager.cc:1397] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260808 13:01:54.858271 22449 ts_tablet_manager.cc:1397] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260808 13:01:54.865232 22447 raft_consensus.cc:348] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b407bd25892c4f0f948b48424ff93307" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38117 } } peers { permanent_uuid: "d73251e095b8425e8ca13eb5d96aedf1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 42759 } } peers { permanent_uuid: "28d923934ea54667a508d559a6388efe" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 35587 } }
I20260808 13:01:54.865875 22447 raft_consensus.cc:374] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:54.866019 22447 raft_consensus.cc:729] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73251e095b8425e8ca13eb5d96aedf1, State: Initialized, Role: FOLLOWER
I20260808 13:01:54.866693 22447 consensus_queue.cc:260] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b407bd25892c4f0f948b48424ff93307" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38117 } } peers { permanent_uuid: "d73251e095b8425e8ca13eb5d96aedf1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 42759 } } peers { permanent_uuid: "28d923934ea54667a508d559a6388efe" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 35587 } }
I20260808 13:01:54.867810 22446 raft_consensus.cc:348] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b407bd25892c4f0f948b48424ff93307" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38117 } } peers { permanent_uuid: "d73251e095b8425e8ca13eb5d96aedf1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 42759 } } peers { permanent_uuid: "28d923934ea54667a508d559a6388efe" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 35587 } }
I20260808 13:01:54.868340 22446 raft_consensus.cc:374] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:54.868487 22446 raft_consensus.cc:729] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d923934ea54667a508d559a6388efe, State: Initialized, Role: FOLLOWER
I20260808 13:01:54.868724 22237 heartbeater.cc:503] Master 127.24.167.62:44085 was elected leader, sending a full tablet report...
I20260808 13:01:54.869024 22449 raft_consensus.cc:348] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b407bd25892c4f0f948b48424ff93307" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38117 } } peers { permanent_uuid: "d73251e095b8425e8ca13eb5d96aedf1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 42759 } } peers { permanent_uuid: "28d923934ea54667a508d559a6388efe" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 35587 } }
I20260808 13:01:54.869506 22447 ts_tablet_manager.cc:1428] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260808 13:01:54.869313 22446 consensus_queue.cc:260] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b407bd25892c4f0f948b48424ff93307" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38117 } } peers { permanent_uuid: "d73251e095b8425e8ca13eb5d96aedf1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 42759 } } peers { permanent_uuid: "28d923934ea54667a508d559a6388efe" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 35587 } }
I20260808 13:01:54.869539 22449 raft_consensus.cc:374] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:54.869630 22449 raft_consensus.cc:729] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b407bd25892c4f0f948b48424ff93307, State: Initialized, Role: FOLLOWER
I20260808 13:01:54.870431 22449 consensus_queue.cc:260] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b407bd25892c4f0f948b48424ff93307" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38117 } } peers { permanent_uuid: "d73251e095b8425e8ca13eb5d96aedf1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 42759 } } peers { permanent_uuid: "28d923934ea54667a508d559a6388efe" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 35587 } }
I20260808 13:01:54.877712 22423 heartbeater.cc:503] Master 127.24.167.62:44085 was elected leader, sending a full tablet report...
I20260808 13:01:54.877895 22446 ts_tablet_manager.cc:1428] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe: Time spent starting tablet: real 0.021s user 0.017s sys 0.003s
I20260808 13:01:54.878643 22449 ts_tablet_manager.cc:1428] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307: Time spent starting tablet: real 0.020s user 0.009s sys 0.012s
I20260808 13:01:54.877667 22457 raft_consensus.cc:482] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 13:01:54.879256 22457 raft_consensus.cc:504] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b407bd25892c4f0f948b48424ff93307" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38117 } } peers { permanent_uuid: "d73251e095b8425e8ca13eb5d96aedf1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 42759 } } peers { permanent_uuid: "28d923934ea54667a508d559a6388efe" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 35587 } }
I20260808 13:01:54.883589 22457 leader_election.cc:288] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b407bd25892c4f0f948b48424ff93307 (127.24.167.3:38117), d73251e095b8425e8ca13eb5d96aedf1 (127.24.167.2:42759)
I20260808 13:01:54.888734 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.882042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:44369 (local address 127.24.167.3:38117)
0808 13:01:54.882258 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:54.882264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:54.882285 (+ 21us) server_negotiation.cc:407] Connection header received
0808 13:01:54.882523 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:54.882528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:54.882597 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:54.882696 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:54.883431 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.884760 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.886801 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.887813 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.887964 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.888015 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:54.888470 (+ 455us) server_negotiation.cc:299] Negotiation successful
0808 13:01:54.888563 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 13:01:54.888950 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.881573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:38117 (local address 127.24.167.1:44369)
0808 13:01:54.882200 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:54.882223 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:54.882364 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:54.882742 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:54.882746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:54.882774 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:54.883172 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:54.883189 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.884977 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.884983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:54.886617 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:54.886630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.887996 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.888007 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:54.888292 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.888330 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 13:01:54.888435 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 13:01:54.888566 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":54,"thread_start_us":1607,"threads_started":1}
I20260808 13:01:54.889894 22352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36eed835df004f76bd10a5e4cd1b814b" candidate_uuid: "28d923934ea54667a508d559a6388efe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b407bd25892c4f0f948b48424ff93307" is_pre_election: true
I20260808 13:01:54.890487 22352 raft_consensus.cc:2454] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28d923934ea54667a508d559a6388efe in term 0.
I20260808 13:01:54.891208 21883 leader_election.cc:302] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28d923934ea54667a508d559a6388efe, b407bd25892c4f0f948b48424ff93307; no voters:
I20260808 13:01:54.891332 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.884171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:41107 (local address 127.24.167.2:42759)
0808 13:01:54.884351 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:54.884356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:54.885359 (+ 1003us) server_negotiation.cc:407] Connection header received
0808 13:01:54.885566 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:54.885572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:54.885651 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:54.885753 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:54.886336 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.887279 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.889586 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.890496 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.890635 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.890737 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:54.891041 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 13:01:54.891152 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 13:01:54.891525 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.883854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:42759 (local address 127.24.167.1:41107)
0808 13:01:54.885254 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:54.885270 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:54.885423 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:54.885768 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:54.885772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:54.885798 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:54.886178 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:54.886187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.887455 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.887458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:54.889377 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:54.889389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.890678 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.890685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:54.890860 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.890901 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 13:01:54.890995 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 13:01:54.891120 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1285,"mutex_wait_us":34,"thread_start_us":1079,"threads_started":1}
I20260808 13:01:54.891662 22457 raft_consensus.cc:2781] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 13:01:54.891924 22457 raft_consensus.cc:482] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 13:01:54.892067 22457 raft_consensus.cc:3037] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:54.892758 22173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36eed835df004f76bd10a5e4cd1b814b" candidate_uuid: "28d923934ea54667a508d559a6388efe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73251e095b8425e8ca13eb5d96aedf1" is_pre_election: true
I20260808 13:01:54.893245 22173 raft_consensus.cc:2454] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28d923934ea54667a508d559a6388efe in term 0.
I20260808 13:01:54.894915 22457 raft_consensus.cc:504] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b407bd25892c4f0f948b48424ff93307" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38117 } } peers { permanent_uuid: "d73251e095b8425e8ca13eb5d96aedf1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 42759 } } peers { permanent_uuid: "28d923934ea54667a508d559a6388efe" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 35587 } }
I20260808 13:01:54.895574 22457 leader_election.cc:288] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [CANDIDATE]: Term 1 election: Requested vote from peers b407bd25892c4f0f948b48424ff93307 (127.24.167.3:38117), d73251e095b8425e8ca13eb5d96aedf1 (127.24.167.2:42759)
I20260808 13:01:54.896008 22352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36eed835df004f76bd10a5e4cd1b814b" candidate_uuid: "28d923934ea54667a508d559a6388efe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b407bd25892c4f0f948b48424ff93307"
I20260808 13:01:54.896073 22173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36eed835df004f76bd10a5e4cd1b814b" candidate_uuid: "28d923934ea54667a508d559a6388efe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73251e095b8425e8ca13eb5d96aedf1"
I20260808 13:01:54.896217 22352 raft_consensus.cc:3037] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:54.896297 22173 raft_consensus.cc:3037] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:54.898785 22352 raft_consensus.cc:2454] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28d923934ea54667a508d559a6388efe in term 1.
I20260808 13:01:54.898808 22173 raft_consensus.cc:2454] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28d923934ea54667a508d559a6388efe in term 1.
I20260808 13:01:54.899209 21883 leader_election.cc:302] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28d923934ea54667a508d559a6388efe, d73251e095b8425e8ca13eb5d96aedf1; no voters:
I20260808 13:01:54.899591 22457 raft_consensus.cc:2781] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:01:54.899904 22457 raft_consensus.cc:686] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [term 1 LEADER]: Becoming Leader. State: Replica: 28d923934ea54667a508d559a6388efe, State: Running, Role: LEADER
I20260808 13:01:54.900388 22457 consensus_queue.cc:237] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b407bd25892c4f0f948b48424ff93307" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38117 } } peers { permanent_uuid: "d73251e095b8425e8ca13eb5d96aedf1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 42759 } } peers { permanent_uuid: "28d923934ea54667a508d559a6388efe" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 35587 } }
I20260808 13:01:54.904667 21775 catalog_manager.cc:5495] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe reported cstate change: term changed from 0 to 1, leader changed from <none> to 28d923934ea54667a508d559a6388efe (127.24.167.1). New cstate: current_term: 1 leader_uuid: "28d923934ea54667a508d559a6388efe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b407bd25892c4f0f948b48424ff93307" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73251e095b8425e8ca13eb5d96aedf1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 42759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28d923934ea54667a508d559a6388efe" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 35587 } health_report { overall_health: HEALTHY } } }
I20260808 13:01:54.927781 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.922024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50106 (local address 127.24.167.1:35587)
0808 13:01:54.922260 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:54.922267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:54.922291 (+ 24us) server_negotiation.cc:407] Connection header received
0808 13:01:54.922458 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:54.922462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:54.922562 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:54.922726 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:01:54.923148 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.923877 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.925255 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.925689 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.925850 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:54.926565 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:01:54.927115 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:01:54.927320 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:54.927482 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 13:01:54.927527 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 13:01:54.936518 22352 raft_consensus.cc:1261] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307 [term 1 FOLLOWER]: Refusing update from remote peer 28d923934ea54667a508d559a6388efe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:01:54.936542 22173 raft_consensus.cc:1261] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1 [term 1 FOLLOWER]: Refusing update from remote peer 28d923934ea54667a508d559a6388efe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:01:54.937495 22457 consensus_queue.cc:1035] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b407bd25892c4f0f948b48424ff93307" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 38117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:01:54.937849 22454 consensus_queue.cc:1035] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73251e095b8425e8ca13eb5d96aedf1" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 42759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:01:54.973085 21775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:01:54.973394 21775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:01:54.977337 21775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4e30afa846b41d683b5eb25a298f4b5: Sending DeleteTablet for 3 replicas of tablet 36eed835df004f76bd10a5e4cd1b814b
I20260808 13:01:54.978734 21951 tablet_service.cc:1514] Processing DeleteTablet for tablet 36eed835df004f76bd10a5e4cd1b814b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:54 UTC) from {username='slave'} at 127.0.0.1:50096
I20260808 13:01:54.978782 22321 tablet_service.cc:1514] Processing DeleteTablet for tablet 36eed835df004f76bd10a5e4cd1b814b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:54 UTC) from {username='slave'} at 127.0.0.1:44282
I20260808 13:01:54.979861 22144 tablet_service.cc:1514] Processing DeleteTablet for tablet 36eed835df004f76bd10a5e4cd1b814b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:54 UTC) from {username='slave'} at 127.0.0.1:55970
I20260808 13:01:54.981608 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 21859
I20260808 13:01:54.991521 22484 tablet_replica.cc:330] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe: stopping tablet replica
I20260808 13:01:54.993156 22484 raft_consensus.cc:2229] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [term 1 LEADER]: Raft consensus shutting down.
I20260808 13:01:54.995676 22485 tablet_replica.cc:330] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1: stopping tablet replica
I20260808 13:01:54.996161 22485 raft_consensus.cc:2229] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:01:54.996837 22485 raft_consensus.cc:2258] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:54.995101 22484 raft_consensus.cc:2258] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:54.998310 22482 tablet_replica.cc:330] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307: stopping tablet replica
I20260808 13:01:54.999013 22482 raft_consensus.cc:2229] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:01:54.999584 22482 raft_consensus.cc:2258] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:54.999603 22485 ts_tablet_manager.cc:1905] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:55.002954 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:54.982545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.24.167.1:35587)
0808 13:01:54.982757 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:54.982766 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:54.993548 (+ 10782us) server_negotiation.cc:407] Connection header received
0808 13:01:54.993855 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:54.993862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:54.993986 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:54.994077 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:54.994867 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.996235 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.998589 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:54.999245 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:54.999349 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:55.001073 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:55.001085 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:55.001105 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:55.001928 (+ 823us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:55.002003 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:55.002031 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:55.002124 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:55.002362 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 13:01:55.002538 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 13:01:55.003718 22482 ts_tablet_manager.cc:1905] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:55.004482 22044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:55.009073 22485 ts_tablet_manager.cc:1918] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:55.009290 22485 log.cc:1196] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/36eed835df004f76bd10a5e4cd1b814b
I20260808 13:01:55.009815 22485 ts_tablet_manager.cc:1939] T 36eed835df004f76bd10a5e4cd1b814b P d73251e095b8425e8ca13eb5d96aedf1: Deleting consensus metadata
I20260808 13:01:55.011651 21756 catalog_manager.cc:4841] TS d73251e095b8425e8ca13eb5d96aedf1 (127.24.167.2:42759): tablet 36eed835df004f76bd10a5e4cd1b814b (table test-table [id=335b9820658e4b19bf1c6a2382738e34]) successfully deleted
I20260808 13:01:55.012933 22482 ts_tablet_manager.cc:1918] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:55.013118 22482 log.cc:1196] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/36eed835df004f76bd10a5e4cd1b814b
I20260808 13:01:55.013659 22482 ts_tablet_manager.cc:1939] T 36eed835df004f76bd10a5e4cd1b814b P b407bd25892c4f0f948b48424ff93307: Deleting consensus metadata
I20260808 13:01:55.015597 21756 catalog_manager.cc:4841] TS b407bd25892c4f0f948b48424ff93307 (127.24.167.3:38117): tablet 36eed835df004f76bd10a5e4cd1b814b (table test-table [id=335b9820658e4b19bf1c6a2382738e34]) successfully deleted
I20260808 13:01:55.004597 22484 ts_tablet_manager.cc:1905] T 36eed835df004f76bd10a5e4cd1b814b P 28d923934ea54667a508d559a6388efe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:55.103536 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 21859
I20260808 13:01:55.130584 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 22058
W20260808 13:01:55.130618 21754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.167.1:35587 (error 108)
I20260808 13:01:55.132764 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:55.132271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:35587 (local address 127.0.0.1:50114)
0808 13:01:55.132511 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:55.132620 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:35587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 13:01:55.133008 21754 catalog_manager.cc:4584] TS 28d923934ea54667a508d559a6388efe (127.24.167.1:35587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36eed835df004f76bd10a5e4cd1b814b: Network error: Client connection negotiation failed: client connection to 127.24.167.1:35587: connect: Connection refused (error 111)
I20260808 13:01:55.138226 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:55.131581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.24.167.2:42759)
0808 13:01:55.131774 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:55.131781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:55.131802 (+ 21us) server_negotiation.cc:407] Connection header received
0808 13:01:55.132060 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:55.132064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:55.132166 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:55.132280 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:55.132714 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:55.133648 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:55.135070 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:55.135509 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:55.135649 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:55.136821 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:55.136830 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:55.136846 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:55.137417 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:55.137477 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:55.137497 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:55.137639 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:55.137770 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 13:01:55.137882 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 13:01:55.139025 22232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:55.186902 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:55.186333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:35587 (local address 127.0.0.1:50116)
0808 13:01:55.186578 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:55.186683 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:35587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260808 13:01:55.188097 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:55.187743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:35587 (local address 127.0.0.1:50130)
0808 13:01:55.187894 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:55.187967 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:35587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 13:01:55.230229 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 22058
I20260808 13:01:55.248890 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:55.248329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:35587 (local address 127.0.0.1:50138)
0808 13:01:55.248582 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:55.248685 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:35587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260808 13:01:55.250006 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:55.249675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:35587 (local address 127.0.0.1:50152)
0808 13:01:55.249816 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:55.249889 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:35587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 13:01:55.258184 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 22240
I20260808 13:01:55.270134 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:55.259100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.24.167.3:38117)
0808 13:01:55.259307 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:55.259314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:55.259337 (+ 23us) server_negotiation.cc:407] Connection header received
0808 13:01:55.259504 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:55.259509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:55.259600 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:55.259717 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:55.260202 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:55.260978 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:55.262362 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:55.262854 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:55.263000 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:55.268662 (+ 5662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:55.268673 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:55.268691 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:01:55.269298 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:55.269358 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:01:55.269378 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:55.269523 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:55.269678 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 13:01:55.269800 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 13:01:55.271248 22414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:55.331214 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:55.330744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:35587 (local address 127.0.0.1:50160)
0808 13:01:55.330944 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:55.331043 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:35587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 13:01:55.332335 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:55.331897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:35587 (local address 127.0.0.1:50162)
0808 13:01:55.332151 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:55.332225 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:35587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260808 13:01:55.360949 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 22240
I20260808 13:01:55.385643 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 21729
I20260808 13:01:55.387023 21814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:55.502872 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 21729
2026-08-08T13:01:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2869 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:01:55 dist-test-slave-tk9z krb5kdc[22501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:01:55 dist-test-slave-tk9z krb5kdc[22501](info): set up 2 sockets
Aug 08 13:01:55 dist-test-slave-tk9z krb5kdc[22501](info): commencing operation
krb5kdc: starting...
W20260808 13:01:56.841935 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.283s user 0.008s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:01:57 dist-test-slave-tk9z krb5kdc[22501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:01:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:01:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 13:01:57.157063 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:45981
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:45981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:01:57.508255 22518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:57.508535 22518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:57.508653 22518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:57.515776 22518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:01:57.515882 22518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:57.515944 22518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:57.516038 22518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:01:57.516124 22518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:01:57.525995 22518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:45981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:45981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:57.527469 22518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:57.528803 22518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:57.540710 22526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:57.540784 22527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:57.540969 22530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:57.541966 22518 server_base.cc:1034] running on GCE node
I20260808 13:01:57.542789 22518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:57.544166 22518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:57.545358 22518 hybrid_clock.cc:648] HybridClock initialized: now 1786194117545296 us; error 54 us; skew 500 ppm
I20260808 13:01:57.545886 22518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:57 dist-test-slave-tk9z krb5kdc[22501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194117, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:57.549151 22518 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:01:57.550722 22518 webserver.cc:460] Webserver started at http://127.24.167.62:46683/ using document root <none> and password file <none>
I20260808 13:01:57.551394 22518 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:57.551508 22518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:57.551772 22518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:57.554824 22518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "094fb997fdca49d4836d29ae12c42769"
format_stamp: "Formatted at 2026-08-08 13:01:57 on dist-test-slave-tk9z"
server_key: "bebe377eafe2ac49c457eaba3c7a5ea0"
server_key_iv: "a672fbc96c40835ff209525d847aadee"
server_key_version: "encryptionkey@0"
I20260808 13:01:57.555553 22518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "094fb997fdca49d4836d29ae12c42769"
format_stamp: "Formatted at 2026-08-08 13:01:57 on dist-test-slave-tk9z"
server_key: "bebe377eafe2ac49c457eaba3c7a5ea0"
server_key_iv: "a672fbc96c40835ff209525d847aadee"
server_key_version: "encryptionkey@0"
I20260808 13:01:57.560734 22518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 13:01:57.564093 22545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:57.565368 22518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 13:01:57.565526 22518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "094fb997fdca49d4836d29ae12c42769"
format_stamp: "Formatted at 2026-08-08 13:01:57 on dist-test-slave-tk9z"
server_key: "bebe377eafe2ac49c457eaba3c7a5ea0"
server_key_iv: "a672fbc96c40835ff209525d847aadee"
server_key_version: "encryptionkey@0"
I20260808 13:01:57.565692 22518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:57.593302 22518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:57.597862 22518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:57.598191 22518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:57.616977 22518 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:45981
I20260808 13:01:57.617002 22628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:45981 every 8 connection(s)
I20260808 13:01:57.618449 22518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:01:57.618896 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 22518
I20260808 13:01:57.619127 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:01:57.619866 25244 external_mini_cluster.cc:1428] Setting key 94941d5485c88663ee7dc0901650748a
I20260808 13:01:57.623795 22629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 13:01:57 dist-test-slave-tk9z krb5kdc[22501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:01:57.638111 22629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769: Bootstrap starting.
I20260808 13:01:57.642956 22629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:57.644367 22629 log.cc:824] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:57.647671 22629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769: No bootstrap required, opened a new log
I20260808 13:01:57.647996 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:57.622316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.24.167.62:45981)
0808 13:01:57.623835 (+ 1519us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:57.623854 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:57.623915 (+ 61us) server_negotiation.cc:407] Connection header received
0808 13:01:57.626659 (+ 2744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:57.626689 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:57.627232 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:57.627720 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:57.629101 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:57.631520 (+ 2419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:57.632878 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:57.634079 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:57.634290 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:57.638477 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:57.638493 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:57.638515 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:57.638581 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:57.643025 (+ 4444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:57.643853 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:57.643867 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:57.643877 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:57.644133 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:57.644581 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:57.644586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:57.644589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:57.645417 (+ 828us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:57.645786 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:57.646331 (+ 545us) server_negotiation.cc:299] Negotiation successful
0808 13:01:57.646868 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1205,"thread_start_us":888,"threads_started":1}
I20260808 13:01:57.656656 22629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094fb997fdca49d4836d29ae12c42769" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45981 } }
I20260808 13:01:57.657083 22629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:57.657197 22629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094fb997fdca49d4836d29ae12c42769, State: Initialized, Role: FOLLOWER
I20260808 13:01:57.657750 22629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094fb997fdca49d4836d29ae12c42769" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45981 } }
I20260808 13:01:57.658056 22629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:01:57.658209 22629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:01:57.658413 22629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:57.661101 22629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094fb997fdca49d4836d29ae12c42769" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45981 } }
I20260808 13:01:57.661662 22629 leader_election.cc:302] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 094fb997fdca49d4836d29ae12c42769; no voters:
I20260808 13:01:57.662252 22629 leader_election.cc:288] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:01:57.662412 22635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:01:57.663105 22635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [term 1 LEADER]: Becoming Leader. State: Replica: 094fb997fdca49d4836d29ae12c42769, State: Running, Role: LEADER
I20260808 13:01:57.663620 22635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094fb997fdca49d4836d29ae12c42769" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45981 } }
I20260808 13:01:57.664367 22629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:01:57.666659 22637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 094fb997fdca49d4836d29ae12c42769. Latest consensus state: current_term: 1 leader_uuid: "094fb997fdca49d4836d29ae12c42769" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094fb997fdca49d4836d29ae12c42769" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45981 } } }
I20260808 13:01:57.667163 22637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [sys.catalog]: This master's current role is: LEADER
I20260808 13:01:57.666956 22636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "094fb997fdca49d4836d29ae12c42769" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094fb997fdca49d4836d29ae12c42769" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 45981 } } }
I20260808 13:01:57.667299 22636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769 [sys.catalog]: This master's current role is: LEADER
I20260808 13:01:57.668241 22646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:01:57.673652 22646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:01:57.681761 22646 catalog_manager.cc:1347] Generated new cluster ID: a5edc551bc79474ca9f0c7e85fd44761
I20260808 13:01:57.681892 22646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:01:57.704368 22646 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:01:57.705736 22646 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:01:57.711217 22646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769: Generated new TSK 0
I20260808 13:01:57.712129 22646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.167.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
Entry for principal kudu/127.24.167.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab.
I20260808 13:01:57.792287 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:45981
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:01:58.149878 22667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:58.150190 22667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:58.150404 22667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:58.157855 22667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:58.158061 22667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:58.158272 22667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 13:01:58.168833 22667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.1.keytab
--principal=kudu/127.24.167.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:45981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:58.170344 22667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:58.171708 22667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:58.185081 22676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:58.185081 22675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:58.185703 22678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:58.186015 22667 server_base.cc:1034] running on GCE node
I20260808 13:01:58.186708 22667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:58.187695 22667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:58.188933 22667 hybrid_clock.cc:648] HybridClock initialized: now 1786194118188889 us; error 54 us; skew 500 ppm
I20260808 13:01:58.189266 22667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:58 dist-test-slave-tk9z krb5kdc[22501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194118, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:58.192585 22667 init.cc:377] Logged in from keytab as kudu/127.24.167.1@KRBTEST.COM (short username kudu)
I20260808 13:01:58.194188 22667 webserver.cc:460] Webserver started at http://127.24.167.1:38583/ using document root <none> and password file <none>
I20260808 13:01:58.194964 22667 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:58.195077 22667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:58.195421 22667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:58.198588 22667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "e04fe939853a45f48bd2dbd21c8d27c6"
format_stamp: "Formatted at 2026-08-08 13:01:58 on dist-test-slave-tk9z"
server_key: "0be8727454e4528931e2249473602df7"
server_key_iv: "45879648f6e9da844003fc023146276f"
server_key_version: "encryptionkey@0"
I20260808 13:01:58.199436 22667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "e04fe939853a45f48bd2dbd21c8d27c6"
format_stamp: "Formatted at 2026-08-08 13:01:58 on dist-test-slave-tk9z"
server_key: "0be8727454e4528931e2249473602df7"
server_key_iv: "45879648f6e9da844003fc023146276f"
server_key_version: "encryptionkey@0"
I20260808 13:01:58.204800 22667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 13:01:58.208340 22686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:58.209654 22667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 13:01:58.209847 22667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "e04fe939853a45f48bd2dbd21c8d27c6"
format_stamp: "Formatted at 2026-08-08 13:01:58 on dist-test-slave-tk9z"
server_key: "0be8727454e4528931e2249473602df7"
server_key_iv: "45879648f6e9da844003fc023146276f"
server_key_version: "encryptionkey@0"
I20260808 13:01:58.210062 22667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:58.226903 22667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:58.231479 22667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:58.231796 22667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:58.232960 22667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:58.234344 22667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:58.234477 22667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 13:01:58.234676 22667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:58.234730 22667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:58.269587 22667 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:44113
I20260808 13:01:58.269691 22854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:44113 every 8 connection(s)
I20260808 13:01:58.271170 22667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 13:01:58.277091 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 22667
I20260808 13:01:58.277299 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 13:01:58.277829 25244 external_mini_cluster.cc:1428] Setting key 21c2585e7ece78a31bc80ebe594a07dd
Aug 08 13:01:58 dist-test-slave-tk9z krb5kdc[22501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194118, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.1@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:01:58.291560 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:58.273739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:35695 (local address 127.24.167.62:45981)
0808 13:01:58.274236 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:58.274242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:58.275298 (+ 1056us) server_negotiation.cc:407] Connection header received
0808 13:01:58.277132 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:58.277138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:58.277237 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:58.277365 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:58.279080 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:58.279883 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:58.281439 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:58.281843 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:58.282008 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:58.285742 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:58.285748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:58.285751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:58.285783 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:58.288363 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:58.289100 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:58.289107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:58.289116 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:58.289236 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:58.289687 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:58.289692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:58.289702 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:58.289973 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:58.290136 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:58.290928 (+ 792us) server_negotiation.cc:299] Negotiation successful
0808 13:01:58.291171 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":367,"thread_start_us":279,"threads_started":1}
I20260808 13:01:58.292533 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:58.274329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45981 (local address 127.24.167.1:35695)
0808 13:01:58.275127 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:58.275165 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:58.276846 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:58.277645 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:58.277654 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:58.278250 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:58.278897 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:58.278914 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:58.280070 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:58.280074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:58.281288 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:58.281301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:58.282113 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:58.282122 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:58.282360 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:58.283127 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:01:58.283156 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:58.285520 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:58.288517 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:58.288526 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:58.288542 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:58.288967 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:58.289351 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:58.289356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:58.289360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:58.289563 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:58.290109 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:58.290117 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:58.290531 (+ 414us) client_negotiation.cc:769] Sending connection context
0808 13:01:58.290810 (+ 279us) client_negotiation.cc:240] Negotiation successful
0808 13:01:58.291336 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":279,"threads_started":1}
I20260808 13:01:58.294201 22855 heartbeater.cc:347] Connected to a master server at 127.24.167.62:45981
I20260808 13:01:58.294580 22855 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:58.295560 22855 heartbeater.cc:511] Master 127.24.167.62:45981 requested a full tablet report, sending...
I20260808 13:01:58.297971 22570 ts_manager.cc:194] Registered new tserver with Master: e04fe939853a45f48bd2dbd21c8d27c6 (127.24.167.1:44113)
I20260808 13:01:58.300253 22570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.1@KRBTEST.COM'} at 127.24.167.1:35695
WARNING: no policy specified for kudu/127.24.167.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
Entry for principal kudu/127.24.167.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab.
I20260808 13:01:58.345855 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:45981
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:01:58.699565 22865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:58.699868 22865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:58.699965 22865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:58.707420 22865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:58.707530 22865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:58.707731 22865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 13:01:58.717834 22865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.2.keytab
--principal=kudu/127.24.167.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:45981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:58.719234 22865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:58.720536 22865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:58.734035 22872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:58.734127 22873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:58.734102 22876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:58.734934 22865 server_base.cc:1034] running on GCE node
I20260808 13:01:58.735548 22865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:58.736506 22865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:58.737753 22865 hybrid_clock.cc:648] HybridClock initialized: now 1786194118737707 us; error 56 us; skew 500 ppm
I20260808 13:01:58.738106 22865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:58 dist-test-slave-tk9z krb5kdc[22501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194118, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:58.741422 22865 init.cc:377] Logged in from keytab as kudu/127.24.167.2@KRBTEST.COM (short username kudu)
I20260808 13:01:58.743059 22865 webserver.cc:460] Webserver started at http://127.24.167.2:46045/ using document root <none> and password file <none>
I20260808 13:01:58.743763 22865 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:58.743913 22865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:58.744251 22865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:58.747411 22865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "c61b0740774c48b8b0ff5881770ac053"
format_stamp: "Formatted at 2026-08-08 13:01:58 on dist-test-slave-tk9z"
server_key: "f57b7e0ea72cf1ff287842f0d5a68f4a"
server_key_iv: "ab5d860e652dac97e6d5fc2dcb42e5c9"
server_key_version: "encryptionkey@0"
I20260808 13:01:58.748212 22865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "c61b0740774c48b8b0ff5881770ac053"
format_stamp: "Formatted at 2026-08-08 13:01:58 on dist-test-slave-tk9z"
server_key: "f57b7e0ea72cf1ff287842f0d5a68f4a"
server_key_iv: "ab5d860e652dac97e6d5fc2dcb42e5c9"
server_key_version: "encryptionkey@0"
I20260808 13:01:58.754035 22865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 13:01:58.758028 22888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:58.759526 22865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:01:58.759744 22865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "c61b0740774c48b8b0ff5881770ac053"
format_stamp: "Formatted at 2026-08-08 13:01:58 on dist-test-slave-tk9z"
server_key: "f57b7e0ea72cf1ff287842f0d5a68f4a"
server_key_iv: "ab5d860e652dac97e6d5fc2dcb42e5c9"
server_key_version: "encryptionkey@0"
I20260808 13:01:58.759972 22865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:58.774050 22865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:58.778733 22865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:58.779062 22865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:58.780223 22865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:58.781663 22865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:58.781740 22865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:58.781829 22865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:58.781879 22865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:58.817023 22865 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:39157
I20260808 13:01:58.817072 23039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:39157 every 8 connection(s)
I20260808 13:01:58.818490 22865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 13:01:58.818845 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 22865
I20260808 13:01:58.819064 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 13:01:58.819761 25244 external_mini_cluster.cc:1428] Setting key df5154248d06dbd5025268daff8ca560
Aug 08 13:01:58 dist-test-slave-tk9z krb5kdc[22501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194118, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.2@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:01:58.843851 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:58.821498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:56149 (local address 127.24.167.62:45981)
0808 13:01:58.822033 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:58.822040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:58.823350 (+ 1310us) server_negotiation.cc:407] Connection header received
0808 13:01:58.826273 (+ 2923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:58.826277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:58.826351 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:58.826527 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:58.828717 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:58.829557 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:58.832166 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:58.832615 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:58.832732 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:58.837427 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:58.837432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:58.837436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:58.837472 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:58.840270 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:58.841281 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:58.841286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:58.841288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:58.841433 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:58.842001 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:58.842004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:58.842006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:58.842282 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:58.842488 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:58.843379 (+ 891us) server_negotiation.cc:299] Negotiation successful
0808 13:01:58.843571 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":275,"threads_started":1}
I20260808 13:01:58.845558 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:58.822163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45981 (local address 127.24.167.2:56149)
0808 13:01:58.823167 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:58.823216 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:58.825955 (+ 2739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:58.826787 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:58.826801 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:58.827620 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:58.828477 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:58.828506 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:58.829720 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:58.829729 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:58.831991 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:58.832011 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:58.832864 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:58.832874 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:58.833237 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:58.834037 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:01:58.834073 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:58.837165 (+ 3092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:58.840436 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:58.840450 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:58.840470 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:58.841125 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:58.841560 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:58.841568 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:58.841574 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:58.841866 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:58.842443 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:58.842454 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:58.842943 (+ 489us) client_negotiation.cc:769] Sending connection context
0808 13:01:58.843298 (+ 355us) client_negotiation.cc:240] Negotiation successful
0808 13:01:58.843928 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":381,"threads_started":1}
I20260808 13:01:58.847237 23040 heartbeater.cc:347] Connected to a master server at 127.24.167.62:45981
I20260808 13:01:58.847580 23040 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:58.848358 23040 heartbeater.cc:511] Master 127.24.167.62:45981 requested a full tablet report, sending...
I20260808 13:01:58.849939 22570 ts_manager.cc:194] Registered new tserver with Master: c61b0740774c48b8b0ff5881770ac053 (127.24.167.2:39157)
I20260808 13:01:58.850904 22570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.2@KRBTEST.COM'} at 127.24.167.2:56149
WARNING: no policy specified for kudu/127.24.167.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
Entry for principal kudu/127.24.167.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab.
I20260808 13:01:58.895140 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:45981
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:01:59.248572 23054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:01:59.248888 23054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:01:59.249097 23054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:01:59.256598 23054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:01:59.256726 23054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:01:59.256919 23054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 13:01:59.267581 23054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.3.keytab
--principal=kudu/127.24.167.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:45981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:01:59.269193 23054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:01:59.271077 23054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:01:59.286419 23064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:59.286360 23061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:01:59.286463 23060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:01:59.287145 23054 server_base.cc:1034] running on GCE node
I20260808 13:01:59.287832 23054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:01:59.288892 23054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:01:59.290138 23054 hybrid_clock.cc:648] HybridClock initialized: now 1786194119290084 us; error 48 us; skew 500 ppm
I20260808 13:01:59.290570 23054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:01:59 dist-test-slave-tk9z krb5kdc[22501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194119, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:01:59.294066 23054 init.cc:377] Logged in from keytab as kudu/127.24.167.3@KRBTEST.COM (short username kudu)
I20260808 13:01:59.295816 23054 webserver.cc:460] Webserver started at http://127.24.167.3:37589/ using document root <none> and password file <none>
I20260808 13:01:59.296518 23054 fs_manager.cc:359] Metadata directory not provided
I20260808 13:01:59.296626 23054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:01:59.296900 23054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:01:59.300122 23054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "eb5ebf7223b040a0b2fd6de259582fad"
format_stamp: "Formatted at 2026-08-08 13:01:59 on dist-test-slave-tk9z"
server_key: "112dbc02acc4323a947b6af5f66250fb"
server_key_iv: "819b48e1c9997a74f35071e552c874bb"
server_key_version: "encryptionkey@0"
I20260808 13:01:59.300872 23054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "eb5ebf7223b040a0b2fd6de259582fad"
format_stamp: "Formatted at 2026-08-08 13:01:59 on dist-test-slave-tk9z"
server_key: "112dbc02acc4323a947b6af5f66250fb"
server_key_iv: "819b48e1c9997a74f35071e552c874bb"
server_key_version: "encryptionkey@0"
I20260808 13:01:59.304199 22855 heartbeater.cc:503] Master 127.24.167.62:45981 was elected leader, sending a full tablet report...
I20260808 13:01:59.306473 23054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260808 13:01:59.310093 23072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:59.311691 23054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:01:59.311875 23054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "eb5ebf7223b040a0b2fd6de259582fad"
format_stamp: "Formatted at 2026-08-08 13:01:59 on dist-test-slave-tk9z"
server_key: "112dbc02acc4323a947b6af5f66250fb"
server_key_iv: "819b48e1c9997a74f35071e552c874bb"
server_key_version: "encryptionkey@0"
I20260808 13:01:59.312093 23054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:01:59.332250 23054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:01:59.336856 23054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:01:59.337184 23054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:01:59.338232 23054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:01:59.339682 23054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:01:59.339759 23054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:59.339919 23054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:01:59.339972 23054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:01:59.376717 23054 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:39085
I20260808 13:01:59.376760 23214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:39085 every 8 connection(s)
I20260808 13:01:59.378157 23054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 13:01:59.381621 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 23054
I20260808 13:01:59.381835 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 13:01:59.382496 25244 external_mini_cluster.cc:1428] Setting key 3b07962886ee1810be5140dfdc487ad1
Aug 08 13:01:59 dist-test-slave-tk9z krb5kdc[22501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194119, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.3@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:01:59.399925 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.380881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:54315 (local address 127.24.167.62:45981)
0808 13:01:59.381435 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:59.381453 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:59.382654 (+ 1201us) server_negotiation.cc:407] Connection header received
0808 13:01:59.385148 (+ 2494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:59.385154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:59.385262 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:59.385385 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:59.387134 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.387902 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.389445 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.389863 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.389967 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.393493 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:59.393498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:59.393502 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:59.393536 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:59.396321 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:59.397366 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:59.397370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:59.397372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:59.397600 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:59.398115 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:59.398122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:59.398125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:59.398577 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:59.398703 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:59.399455 (+ 752us) server_negotiation.cc:299] Negotiation successful
0808 13:01:59.399644 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":220,"threads_started":1}
I20260808 13:01:59.400938 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.381459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:45981 (local address 127.24.167.3:54315)
0808 13:01:59.382456 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:59.382524 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:59.384875 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:59.385711 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:59.385723 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:59.386315 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:01:59.386976 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:59.386993 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.388042 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.388046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:59.389292 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:59.389304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.390067 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.390072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:59.390286 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.391016 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:01:59.391044 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:01:59.393290 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:01:59.396516 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:59.396526 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:59.396546 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:59.397225 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:59.397729 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:01:59.397734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:01:59.397737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:01:59.397984 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:01:59.398710 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:01:59.398716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:01:59.399081 (+ 365us) client_negotiation.cc:769] Sending connection context
0808 13:01:59.399328 (+ 247us) client_negotiation.cc:240] Negotiation successful
0808 13:01:59.399774 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":395,"threads_started":1}
I20260808 13:01:59.402415 23215 heartbeater.cc:347] Connected to a master server at 127.24.167.62:45981
I20260808 13:01:59.402743 23215 heartbeater.cc:464] Registering TS with master...
I20260808 13:01:59.403502 23215 heartbeater.cc:511] Master 127.24.167.62:45981 requested a full tablet report, sending...
I20260808 13:01:59.404909 22570 ts_manager.cc:194] Registered new tserver with Master: eb5ebf7223b040a0b2fd6de259582fad (127.24.167.3:39085)
I20260808 13:01:59.405742 22570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.167.3@KRBTEST.COM'} at 127.24.167.3:54315
I20260808 13:01:59.410811 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 13:01:59.428638 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.415361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.24.167.62:45981)
0808 13:01:59.415575 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:59.415583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:59.416002 (+ 419us) server_negotiation.cc:407] Connection header received
0808 13:01:59.416188 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:59.416192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:59.416257 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:59.416372 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:59.417950 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.418784 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.420044 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.420434 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.420529 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.422920 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:59.422925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:59.422928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:59.422966 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:59.425779 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:59.426492 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:59.426496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:59.426498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:59.426631 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:59.427063 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:59.427066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:59.427068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:59.427413 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:59.427618 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:59.427945 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 13:01:59.428211 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 13:01:59.433229 22570 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 13:01:59.440325 22570 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 13:01:59.477136 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.453075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.24.167.1:44113)
0808 13:01:59.466713 (+ 13638us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:59.466726 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:59.466764 (+ 38us) server_negotiation.cc:407] Connection header received
0808 13:01:59.466900 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:59.466911 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:59.467205 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:59.469220 (+ 2015us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:59.469512 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.471213 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.474172 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.475683 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.475853 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.476122 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:59.476632 (+ 510us) server_negotiation.cc:299] Negotiation successful
0808 13:01:59.476801 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13408,"thread_start_us":13194,"threads_started":1}
I20260808 13:01:59.477173 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.452744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:44113 (local address 127.0.0.1:40232)
0808 13:01:59.465437 (+ 12693us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:59.465509 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:59.465868 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:59.468357 (+ 2489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:59.468367 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:59.468399 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:59.468932 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:59.468946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.471366 (+ 2420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.471371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:59.473959 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:59.473978 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.475926 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.475935 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:59.476346 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.476423 (+ 77us) client_negotiation.cc:769] Sending connection context
0808 13:01:59.476560 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 13:01:59.476751 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12310,"mutex_wait_us":90,"thread_start_us":14772,"threads_started":1}
I20260808 13:01:59.479183 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.453387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.24.167.3:39085)
0808 13:01:59.464603 (+ 11216us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:59.464618 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:59.467387 (+ 2769us) server_negotiation.cc:407] Connection header received
0808 13:01:59.467581 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:59.467588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:59.467801 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:59.467999 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:59.469640 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.472686 (+ 3046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.476531 (+ 3845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.477946 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.478079 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.478330 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:59.478639 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 13:01:59.478840 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10986,"thread_start_us":10592,"threads_started":1}
I20260808 13:01:59.479956 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.452970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:39085 (local address 127.0.0.1:33370)
0808 13:01:59.467263 (+ 14293us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:59.467281 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:59.467428 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:59.468891 (+ 1463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:59.468897 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:59.468925 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:59.469417 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:59.469431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.472867 (+ 3436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.472872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:59.476360 (+ 3488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:59.476375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.478167 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.478173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:59.478361 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.478432 (+ 71us) client_negotiation.cc:769] Sending connection context
0808 13:01:59.479400 (+ 968us) client_negotiation.cc:240] Negotiation successful
0808 13:01:59.479553 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14161,"mutex_wait_us":1074,"thread_start_us":11917,"threads_started":1}
I20260808 13:01:59.481269 22753 tablet_service.cc:1467] Processing CreateTablet for tablet 584ca0240dad4e13b8637357d86ca6dd (DEFAULT_TABLE table=test-table [id=1b881c7b62b0468bb2c13973fa205a28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:59.482640 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.466707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:39157 (local address 127.0.0.1:39672)
0808 13:01:59.474251 (+ 7544us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:59.474269 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:59.474452 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:59.475602 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:59.475611 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:59.475638 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:59.476136 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:59.476152 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.478290 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.478297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:59.480428 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:59.480447 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.481789 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.481796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:59.481995 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.482032 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 13:01:59.482129 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 13:01:59.482291 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7414,"thread_start_us":7239,"threads_started":1}
I20260808 13:01:59.482717 22753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584ca0240dad4e13b8637357d86ca6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:59.483170 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.466333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39672 (local address 127.24.167.2:39157)
0808 13:01:59.474617 (+ 8284us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:59.474634 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:59.474685 (+ 51us) server_negotiation.cc:407] Connection header received
0808 13:01:59.474906 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:59.474915 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:59.475150 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:59.475351 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:59.476340 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.478108 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.480589 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.481575 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.481742 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.481938 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:59.482661 (+ 723us) server_negotiation.cc:299] Negotiation successful
0808 13:01:59.482850 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8140,"thread_start_us":7986,"threads_started":1}
I20260808 13:01:59.485855 23133 tablet_service.cc:1467] Processing CreateTablet for tablet 584ca0240dad4e13b8637357d86ca6dd (DEFAULT_TABLE table=test-table [id=1b881c7b62b0468bb2c13973fa205a28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:59.486882 22955 tablet_service.cc:1467] Processing CreateTablet for tablet 584ca0240dad4e13b8637357d86ca6dd (DEFAULT_TABLE table=test-table [id=1b881c7b62b0468bb2c13973fa205a28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:01:59.487493 23133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584ca0240dad4e13b8637357d86ca6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:59.488234 22955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584ca0240dad4e13b8637357d86ca6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:01:59.494397 23256 tablet_bootstrap.cc:492] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6: Bootstrap starting.
I20260808 13:01:59.498047 23256 tablet_bootstrap.cc:654] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:59.499168 23256 log.cc:824] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:59.501940 23256 tablet_bootstrap.cc:492] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6: No bootstrap required, opened a new log
I20260808 13:01:59.502349 23256 ts_tablet_manager.cc:1397] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260808 13:01:59.502669 23259 tablet_bootstrap.cc:492] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053: Bootstrap starting.
I20260808 13:01:59.503811 23260 tablet_bootstrap.cc:492] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad: Bootstrap starting.
I20260808 13:01:59.506819 23260 tablet_bootstrap.cc:654] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:59.507532 23259 tablet_bootstrap.cc:654] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053: Neither blocks nor log segments found. Creating new log.
I20260808 13:01:59.507906 23260 log.cc:824] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:59.508720 23259 log.cc:824] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053: Log is configured to *not* fsync() on all Append() calls
I20260808 13:01:59.510975 23260 tablet_bootstrap.cc:492] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad: No bootstrap required, opened a new log
I20260808 13:01:59.511250 23260 ts_tablet_manager.cc:1397] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 13:01:59.511742 23259 tablet_bootstrap.cc:492] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053: No bootstrap required, opened a new log
I20260808 13:01:59.512037 23259 ts_tablet_manager.cc:1397] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260808 13:01:59.513885 23256 raft_consensus.cc:348] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04fe939853a45f48bd2dbd21c8d27c6" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44113 } } peers { permanent_uuid: "eb5ebf7223b040a0b2fd6de259582fad" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39085 } } peers { permanent_uuid: "c61b0740774c48b8b0ff5881770ac053" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39157 } }
I20260808 13:01:59.514340 23256 raft_consensus.cc:374] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:59.514508 23256 raft_consensus.cc:729] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04fe939853a45f48bd2dbd21c8d27c6, State: Initialized, Role: FOLLOWER
I20260808 13:01:59.515169 23256 consensus_queue.cc:260] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04fe939853a45f48bd2dbd21c8d27c6" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44113 } } peers { permanent_uuid: "eb5ebf7223b040a0b2fd6de259582fad" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39085 } } peers { permanent_uuid: "c61b0740774c48b8b0ff5881770ac053" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39157 } }
I20260808 13:01:59.517038 23256 ts_tablet_manager.cc:1428] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260808 13:01:59.521278 23260 raft_consensus.cc:348] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04fe939853a45f48bd2dbd21c8d27c6" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44113 } } peers { permanent_uuid: "eb5ebf7223b040a0b2fd6de259582fad" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39085 } } peers { permanent_uuid: "c61b0740774c48b8b0ff5881770ac053" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39157 } }
I20260808 13:01:59.521418 23259 raft_consensus.cc:348] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04fe939853a45f48bd2dbd21c8d27c6" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44113 } } peers { permanent_uuid: "eb5ebf7223b040a0b2fd6de259582fad" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39085 } } peers { permanent_uuid: "c61b0740774c48b8b0ff5881770ac053" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39157 } }
I20260808 13:01:59.521766 23260 raft_consensus.cc:374] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:59.521890 23260 raft_consensus.cc:729] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5ebf7223b040a0b2fd6de259582fad, State: Initialized, Role: FOLLOWER
I20260808 13:01:59.521937 23259 raft_consensus.cc:374] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:01:59.522086 23259 raft_consensus.cc:729] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61b0740774c48b8b0ff5881770ac053, State: Initialized, Role: FOLLOWER
I20260808 13:01:59.522576 23260 consensus_queue.cc:260] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04fe939853a45f48bd2dbd21c8d27c6" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44113 } } peers { permanent_uuid: "eb5ebf7223b040a0b2fd6de259582fad" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39085 } } peers { permanent_uuid: "c61b0740774c48b8b0ff5881770ac053" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39157 } }
I20260808 13:01:59.522711 23259 consensus_queue.cc:260] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04fe939853a45f48bd2dbd21c8d27c6" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44113 } } peers { permanent_uuid: "eb5ebf7223b040a0b2fd6de259582fad" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39085 } } peers { permanent_uuid: "c61b0740774c48b8b0ff5881770ac053" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39157 } }
W20260808 13:01:59.523545 22857 tablet.cc:2369] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:01:59.523785 23215 heartbeater.cc:503] Master 127.24.167.62:45981 was elected leader, sending a full tablet report...
I20260808 13:01:59.524528 23260 ts_tablet_manager.cc:1428] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260808 13:01:59.525235 23040 heartbeater.cc:503] Master 127.24.167.62:45981 was elected leader, sending a full tablet report...
I20260808 13:01:59.525701 23259 ts_tablet_manager.cc:1428] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
W20260808 13:01:59.570729 23046 tablet.cc:2369] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 13:01:59.629951 23219 tablet.cc:2369] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:01:59.661417 23266 raft_consensus.cc:482] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 13:01:59.661774 23266 raft_consensus.cc:504] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04fe939853a45f48bd2dbd21c8d27c6" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44113 } } peers { permanent_uuid: "eb5ebf7223b040a0b2fd6de259582fad" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39085 } } peers { permanent_uuid: "c61b0740774c48b8b0ff5881770ac053" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39157 } }
I20260808 13:01:59.663703 23266 leader_election.cc:288] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e04fe939853a45f48bd2dbd21c8d27c6 (127.24.167.1:44113), eb5ebf7223b040a0b2fd6de259582fad (127.24.167.3:39085)
I20260808 13:01:59.671389 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.664996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:59479 (local address 127.24.167.1:44113)
0808 13:01:59.665195 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:59.665200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:59.665324 (+ 124us) server_negotiation.cc:407] Connection header received
0808 13:01:59.665762 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:59.665770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:59.665854 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:59.665971 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:59.666552 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.667363 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.669393 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.670442 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.670634 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.670717 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:59.670997 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 13:01:59.671128 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 13:01:59.671521 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.663920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:44113 (local address 127.24.167.2:59479)
0808 13:01:59.665128 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:59.665154 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:59.665480 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:59.665994 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:59.665998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:59.666025 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:59.666347 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:59.666356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.667508 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.667514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:59.669236 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:59.669249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.670666 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.670672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:59.670826 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.670850 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 13:01:59.670974 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 13:01:59.671111 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":707,"threads_started":1}
I20260808 13:01:59.671732 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.664339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:54777 (local address 127.24.167.3:39085)
0808 13:01:59.664594 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:59.664600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:59.665872 (+ 1272us) server_negotiation.cc:407] Connection header received
0808 13:01:59.666176 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:59.666180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:59.666241 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:59.666341 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:01:59.666926 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.667669 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.669880 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.670750 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.670926 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.671004 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:59.671333 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 13:01:59.671477 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 13:01:59.671840 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.664238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:39085 (local address 127.24.167.2:54777)
0808 13:01:59.665714 (+ 1476us) negotiation.cc:107] Waiting for socket to connect
0808 13:01:59.665733 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 13:01:59.665947 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:01:59.666363 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:01:59.666390 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:01:59.666421 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:01:59.666775 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:59.666783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.667819 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.667825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:59.669680 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:01:59.669695 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.670992 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.670998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:01:59.671168 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.671191 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 13:01:59.671306 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 13:01:59.671428 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"mutex_wait_us":4,"thread_start_us":1013,"threads_started":1}
I20260808 13:01:59.672869 22781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584ca0240dad4e13b8637357d86ca6dd" candidate_uuid: "c61b0740774c48b8b0ff5881770ac053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04fe939853a45f48bd2dbd21c8d27c6" is_pre_election: true
I20260808 13:01:59.672892 23155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584ca0240dad4e13b8637357d86ca6dd" candidate_uuid: "c61b0740774c48b8b0ff5881770ac053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb5ebf7223b040a0b2fd6de259582fad" is_pre_election: true
I20260808 13:01:59.673427 22781 raft_consensus.cc:2454] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c61b0740774c48b8b0ff5881770ac053 in term 0.
I20260808 13:01:59.673444 23155 raft_consensus.cc:2454] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c61b0740774c48b8b0ff5881770ac053 in term 0.
I20260808 13:01:59.674154 22894 leader_election.cc:302] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c61b0740774c48b8b0ff5881770ac053, eb5ebf7223b040a0b2fd6de259582fad; no voters:
I20260808 13:01:59.674641 23266 raft_consensus.cc:2781] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 13:01:59.674786 23266 raft_consensus.cc:482] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 13:01:59.674898 23266 raft_consensus.cc:3037] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:59.677134 23266 raft_consensus.cc:504] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04fe939853a45f48bd2dbd21c8d27c6" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44113 } } peers { permanent_uuid: "eb5ebf7223b040a0b2fd6de259582fad" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39085 } } peers { permanent_uuid: "c61b0740774c48b8b0ff5881770ac053" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39157 } }
I20260808 13:01:59.677879 23266 leader_election.cc:288] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [CANDIDATE]: Term 1 election: Requested vote from peers e04fe939853a45f48bd2dbd21c8d27c6 (127.24.167.1:44113), eb5ebf7223b040a0b2fd6de259582fad (127.24.167.3:39085)
I20260808 13:01:59.678323 22781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584ca0240dad4e13b8637357d86ca6dd" candidate_uuid: "c61b0740774c48b8b0ff5881770ac053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04fe939853a45f48bd2dbd21c8d27c6"
I20260808 13:01:59.678465 23155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584ca0240dad4e13b8637357d86ca6dd" candidate_uuid: "c61b0740774c48b8b0ff5881770ac053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb5ebf7223b040a0b2fd6de259582fad"
I20260808 13:01:59.678596 22781 raft_consensus.cc:3037] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:59.678675 23155 raft_consensus.cc:3037] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:01:59.681869 22781 raft_consensus.cc:2454] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c61b0740774c48b8b0ff5881770ac053 in term 1.
I20260808 13:01:59.681869 23155 raft_consensus.cc:2454] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c61b0740774c48b8b0ff5881770ac053 in term 1.
I20260808 13:01:59.682279 22894 leader_election.cc:302] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c61b0740774c48b8b0ff5881770ac053, eb5ebf7223b040a0b2fd6de259582fad; no voters:
I20260808 13:01:59.682588 23266 raft_consensus.cc:2781] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:01:59.683032 23266 raft_consensus.cc:686] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [term 1 LEADER]: Becoming Leader. State: Replica: c61b0740774c48b8b0ff5881770ac053, State: Running, Role: LEADER
I20260808 13:01:59.683507 23266 consensus_queue.cc:237] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04fe939853a45f48bd2dbd21c8d27c6" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44113 } } peers { permanent_uuid: "eb5ebf7223b040a0b2fd6de259582fad" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39085 } } peers { permanent_uuid: "c61b0740774c48b8b0ff5881770ac053" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39157 } }
I20260808 13:01:59.688246 22569 catalog_manager.cc:5495] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 reported cstate change: term changed from 0 to 1, leader changed from <none> to c61b0740774c48b8b0ff5881770ac053 (127.24.167.2). New cstate: current_term: 1 leader_uuid: "c61b0740774c48b8b0ff5881770ac053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04fe939853a45f48bd2dbd21c8d27c6" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb5ebf7223b040a0b2fd6de259582fad" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c61b0740774c48b8b0ff5881770ac053" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 39157 } health_report { overall_health: HEALTHY } } }
I20260808 13:01:59.723850 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.718290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.24.167.2:39157)
0808 13:01:59.718521 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:59.718526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:59.718650 (+ 124us) server_negotiation.cc:407] Connection header received
0808 13:01:59.718828 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:59.718832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:59.718916 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:59.719063 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:01:59.719410 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.720175 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.721614 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.722066 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.722165 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.722772 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:01:59.723338 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:01:59.723465 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:59.723552 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 13:01:59.723627 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 13:01:59.732823 23155 raft_consensus.cc:1261] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad [term 1 FOLLOWER]: Refusing update from remote peer c61b0740774c48b8b0ff5881770ac053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:01:59.732854 22781 raft_consensus.cc:1261] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6 [term 1 FOLLOWER]: Refusing update from remote peer c61b0740774c48b8b0ff5881770ac053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:01:59.733896 23269 consensus_queue.cc:1035] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5ebf7223b040a0b2fd6de259582fad" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 39085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:01:59.734278 23266 consensus_queue.cc:1035] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e04fe939853a45f48bd2dbd21c8d27c6" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:01:59.745829 23272 mvcc.cc:205] Tried to move back new op lower bound from 7316251114414116864 to 7316251114227126272. Current Snapshot: MvccSnapshot[applied={T|T < 7316251114414116864}]
I20260808 13:01:59.747191 23274 mvcc.cc:205] Tried to move back new op lower bound from 7316251114414116864 to 7316251114227126272. Current Snapshot: MvccSnapshot[applied={T|T < 7316251114414116864}]
I20260808 13:01:59.751655 23275 mvcc.cc:205] Tried to move back new op lower bound from 7316251114414116864 to 7316251114227126272. Current Snapshot: MvccSnapshot[applied={T|T < 7316251114414116864}]
I20260808 13:01:59.760792 22569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:01:59.761083 22569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:01:59.765079 22569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 094fb997fdca49d4836d29ae12c42769: Sending DeleteTablet for 3 replicas of tablet 584ca0240dad4e13b8637357d86ca6dd
I20260808 13:01:59.766476 22753 tablet_service.cc:1514] Processing DeleteTablet for tablet 584ca0240dad4e13b8637357d86ca6dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:59 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:40232
I20260808 13:01:59.766500 23133 tablet_service.cc:1514] Processing DeleteTablet for tablet 584ca0240dad4e13b8637357d86ca6dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:59 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:33370
I20260808 13:01:59.767603 22955 tablet_service.cc:1514] Processing DeleteTablet for tablet 584ca0240dad4e13b8637357d86ca6dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:01:59 UTC) from {username='kudu', principal='kudu/127.24.167.62@KRBTEST.COM'} at 127.0.0.1:39672
I20260808 13:01:59.768889 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 22667
I20260808 13:01:59.779482 23289 tablet_replica.cc:330] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad: stopping tablet replica
I20260808 13:01:59.780388 23289 raft_consensus.cc:2229] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:01:59.781325 23289 raft_consensus.cc:2258] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:59.784340 23289 ts_tablet_manager.cc:1905] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:59.787057 23291 tablet_replica.cc:330] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053: stopping tablet replica
I20260808 13:01:59.787703 23291 raft_consensus.cc:2229] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [term 1 LEADER]: Raft consensus shutting down.
I20260808 13:01:59.791989 23289 ts_tablet_manager.cc:1918] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:59.791968 23291 raft_consensus.cc:2258] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:01:59.792246 23289 log.cc:1196] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/584ca0240dad4e13b8637357d86ca6dd
I20260808 13:01:59.793146 23289 ts_tablet_manager.cc:1939] T 584ca0240dad4e13b8637357d86ca6dd P eb5ebf7223b040a0b2fd6de259582fad: Deleting consensus metadata
I20260808 13:01:59.795835 23291 ts_tablet_manager.cc:1905] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:59.795971 22549 catalog_manager.cc:4841] TS eb5ebf7223b040a0b2fd6de259582fad (127.24.167.3:39085): tablet 584ca0240dad4e13b8637357d86ca6dd (table test-table [id=1b881c7b62b0468bb2c13973fa205a28]) successfully deleted
I20260808 13:01:59.796574 23288 tablet_replica.cc:330] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6: stopping tablet replica
I20260808 13:01:59.796986 23288 raft_consensus.cc:2229] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:01:59.798929 23288 raft_consensus.cc:2258] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 08 13:01:59 dist-test-slave-tk9z krb5kdc[22501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.1@KRBTEST.COM
I20260808 13:01:59.801380 23288 ts_tablet_manager.cc:1905] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:01:59.807766 23291 ts_tablet_manager.cc:1918] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:59.808055 23291 log.cc:1196] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/584ca0240dad4e13b8637357d86ca6dd
I20260808 13:01:59.808774 23291 ts_tablet_manager.cc:1939] T 584ca0240dad4e13b8637357d86ca6dd P c61b0740774c48b8b0ff5881770ac053: Deleting consensus metadata
I20260808 13:01:59.808945 23288 ts_tablet_manager.cc:1918] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:01:59.809792 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.770698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40248 (local address 127.24.167.1:44113)
0808 13:01:59.770935 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:59.770955 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:59.776186 (+ 5231us) server_negotiation.cc:407] Connection header received
0808 13:01:59.776404 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:59.776412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:59.777939 (+ 1527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:59.778115 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:59.788033 (+ 9918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.789316 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.792421 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.793200 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.793308 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.801418 (+ 8110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:59.801438 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:59.801458 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:59.801510 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:59.805762 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:59.806806 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:59.806815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:59.806819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:59.807027 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:59.807533 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:59.807538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:59.807541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:59.808069 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:59.808406 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:59.809090 (+ 684us) server_negotiation.cc:299] Negotiation successful
0808 13:01:59.809343 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 13:01:59.809459 23288 log.cc:1196] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786193784050718-25244-0/minicluster-data/ts-0/wal/wals/584ca0240dad4e13b8637357d86ca6dd
I20260808 13:01:59.810902 23288 ts_tablet_manager.cc:1939] T 584ca0240dad4e13b8637357d86ca6dd P e04fe939853a45f48bd2dbd21c8d27c6: Deleting consensus metadata
I20260808 13:01:59.810861 22549 catalog_manager.cc:4841] TS c61b0740774c48b8b0ff5881770ac053 (127.24.167.2:39157): tablet 584ca0240dad4e13b8637357d86ca6dd (table test-table [id=1b881c7b62b0468bb2c13973fa205a28]) successfully deleted
I20260808 13:01:59.810926 22849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:01:59.935848 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 22667
I20260808 13:01:59.935940 22546 catalog_manager.cc:4841] TS e04fe939853a45f48bd2dbd21c8d27c6 (127.24.167.1:44113): tablet 584ca0240dad4e13b8637357d86ca6dd (table test-table [id=1b881c7b62b0468bb2c13973fa205a28]) successfully deleted
I20260808 13:01:59.962114 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 22865
Aug 08 13:01:59 dist-test-slave-tk9z krb5kdc[22501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.2@KRBTEST.COM
I20260808 13:01:59.979561 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:01:59.963011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.24.167.2:39157)
0808 13:01:59.963193 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 13:01:59.963200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:01:59.963241 (+ 41us) server_negotiation.cc:407] Connection header received
0808 13:01:59.963484 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:01:59.963490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:01:59.963577 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:01:59.963697 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:01:59.965536 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.966423 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.967733 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:01:59.968265 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:01:59.968402 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:01:59.973106 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:01:59.973122 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:01:59.973139 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:01:59.973181 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:01:59.976508 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:59.977272 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:59.977289 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:59.977291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:59.977450 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:01:59.977927 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:01:59.977938 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:01:59.977942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:01:59.978331 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:01:59.978721 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0808 13:01:59.979048 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 13:01:59.979219 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 13:01:59.980352 23033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:00.096879 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 22865
I20260808 13:02:00.124644 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 23054
Aug 08 13:02:00 dist-test-slave-tk9z krb5kdc[22501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.3@KRBTEST.COM
I20260808 13:02:00.140329 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:00.125514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33380 (local address 127.24.167.3:39085)
0808 13:02:00.125744 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:00.125749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:00.125771 (+ 22us) server_negotiation.cc:407] Connection header received
0808 13:02:00.125835 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:00.125839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:00.125904 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:00.125996 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:00.127427 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:00.128172 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:00.129445 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:00.129835 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:00.129931 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:00.134693 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:00.134710 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:00.134729 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:00.134774 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:00.137498 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:00.138176 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:00.138184 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:00.138186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:00.138333 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:00.138901 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:00.138904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:00.138906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:00.139280 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:00.139498 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:00.139805 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 13:02:00.140009 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260808 13:02:00.141029 23209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:00.231477 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 23054
I20260808 13:02:00.254302 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 22518
I20260808 13:02:00.255640 22619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:00.367401 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 22518
2026-08-08T13:02:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4864 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T13:02:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:00Z Disabled control of system clock
I20260808 13:02:00.475059 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:39575
--webserver_interface=127.24.167.62
--webserver_port=0
--builtin_ntp_servers=127.24.167.20:46269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:39575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 13:02:00.836018 23306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:00.836350 23306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:00.836503 23306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:00.843925 23306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:00.844027 23306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:00.844123 23306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:00.844225 23306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:00.844314 23306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:00.853638 23306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:39575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:39575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:00.855067 23306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:00.856858 23306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:00.869285 23311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:00.869201 23313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:00.869243 23315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:00.870440 23306 server_base.cc:1034] running on GCE node
I20260808 13:02:00.871248 23306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:00.872632 23306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:00.873895 23306 hybrid_clock.cc:648] HybridClock initialized: now 1786194120873835 us; error 46 us; skew 500 ppm
I20260808 13:02:00.874267 23306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 13:02:00.880136 23306 webserver.cc:460] Webserver started at http://127.24.167.62:42585/ using document root <none> and password file <none>
I20260808 13:02:00.880950 23306 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:00.881106 23306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:00.881507 23306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:00.884723 23306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "1975ba4d43f34bdd967d9a5108e658e4"
format_stamp: "Formatted at 2026-08-08 13:02:00 on dist-test-slave-tk9z"
server_key: "755c70d2528fe04238a46bf51bde1d83"
server_key_iv: "516149e6fa4597393d1268b6899d70c0"
server_key_version: "encryptionkey@0"
I20260808 13:02:00.885649 23306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "1975ba4d43f34bdd967d9a5108e658e4"
format_stamp: "Formatted at 2026-08-08 13:02:00 on dist-test-slave-tk9z"
server_key: "755c70d2528fe04238a46bf51bde1d83"
server_key_iv: "516149e6fa4597393d1268b6899d70c0"
server_key_version: "encryptionkey@0"
I20260808 13:02:00.891235 23306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260808 13:02:00.895020 23327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:00.896579 23306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 13:02:00.896801 23306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "1975ba4d43f34bdd967d9a5108e658e4"
format_stamp: "Formatted at 2026-08-08 13:02:00 on dist-test-slave-tk9z"
server_key: "755c70d2528fe04238a46bf51bde1d83"
server_key_iv: "516149e6fa4597393d1268b6899d70c0"
server_key_version: "encryptionkey@0"
I20260808 13:02:00.897065 23306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:00.924873 23306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:00.926132 23306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:00.926553 23306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:00.951143 23306 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:39575
I20260808 13:02:00.951162 23398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:39575 every 8 connection(s)
I20260808 13:02:00.952548 23306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:00.956456 23399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:00.957407 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 23306
I20260808 13:02:00.957612 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:00.958110 25244 external_mini_cluster.cc:1428] Setting key 5f765af878a5ca68128e41df31f437a9
I20260808 13:02:00.970255 23399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4: Bootstrap starting.
I20260808 13:02:00.971922 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:00.960112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.24.167.62:39575)
0808 13:02:00.961140 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:00.961151 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:00.961193 (+ 42us) server_negotiation.cc:407] Connection header received
0808 13:02:00.963200 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:00.963222 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:00.963665 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:00.964071 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:00.964624 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:00.965879 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:00.967315 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:00.968006 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:00.968226 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:00.969629 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:00.969637 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:00.969650 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:00.970190 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:00.970286 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:00.970307 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:00.970467 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:00.970704 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 13:02:00.971009 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":372,"threads_started":1}
I20260808 13:02:00.974210 23399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:00.975660 23399 log.cc:824] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:00.978650 23399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4: No bootstrap required, opened a new log
I20260808 13:02:00.986851 23399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1975ba4d43f34bdd967d9a5108e658e4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39575 } }
I20260808 13:02:00.987306 23399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:00.987423 23399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1975ba4d43f34bdd967d9a5108e658e4, State: Initialized, Role: FOLLOWER
I20260808 13:02:00.988080 23399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1975ba4d43f34bdd967d9a5108e658e4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39575 } }
I20260808 13:02:00.988353 23399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:00.988519 23399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:00.988704 23399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:00.991074 23399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1975ba4d43f34bdd967d9a5108e658e4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39575 } }
I20260808 13:02:00.991604 23399 leader_election.cc:302] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1975ba4d43f34bdd967d9a5108e658e4; no voters:
I20260808 13:02:00.992226 23399 leader_election.cc:288] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:00.992439 23407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:00.993216 23407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [term 1 LEADER]: Becoming Leader. State: Replica: 1975ba4d43f34bdd967d9a5108e658e4, State: Running, Role: LEADER
I20260808 13:02:00.993855 23407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1975ba4d43f34bdd967d9a5108e658e4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39575 } }
I20260808 13:02:00.994597 23399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:00.997045 23408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1975ba4d43f34bdd967d9a5108e658e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1975ba4d43f34bdd967d9a5108e658e4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39575 } } }
I20260808 13:02:00.997038 23409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1975ba4d43f34bdd967d9a5108e658e4. Latest consensus state: current_term: 1 leader_uuid: "1975ba4d43f34bdd967d9a5108e658e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1975ba4d43f34bdd967d9a5108e658e4" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 39575 } } }
I20260808 13:02:00.997537 23408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:00.997538 23409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:01.000370 23417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:01.005998 23417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:01.014581 23417 catalog_manager.cc:1347] Generated new cluster ID: 996d2ab405d94bfb90425021ca90925f
I20260808 13:02:01.014696 23417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:01.039151 23417 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:01.040570 23417 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:01.056310 23417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4: Generated new TSK 0
I20260808 13:02:01.057365 23417 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:01.073721 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.1:0
--local_ip_for_outbound_sockets=127.24.167.1
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:39575
--builtin_ntp_servers=127.24.167.20:46269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 13:02:01.452991 23436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:01.453346 23436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:01.453508 23436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:01.461045 23436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:01.461212 23436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:01.461412 23436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.1
I20260808 13:02:01.471380 23436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.167.1
--webserver_port=0
--tserver_master_addrs=127.24.167.62:39575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.1
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:01.472795 23436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:01.474762 23436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:01.488673 23442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:01.488634 23446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:01.488581 23444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:01.489799 23436 server_base.cc:1034] running on GCE node
I20260808 13:02:01.490526 23436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:01.491513 23436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:01.492723 23436 hybrid_clock.cc:648] HybridClock initialized: now 1786194121492679 us; error 43 us; skew 500 ppm
I20260808 13:02:01.493065 23436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 13:02:01.495704 23436 webserver.cc:460] Webserver started at http://127.24.167.1:39299/ using document root <none> and password file <none>
I20260808 13:02:01.496464 23436 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:01.496590 23436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:01.496932 23436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:01.500156 23436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/data/instance:
uuid: "8eb8ba16bceb4823a23d5214c95ccaa0"
format_stamp: "Formatted at 2026-08-08 13:02:01 on dist-test-slave-tk9z"
server_key: "fbfae1d96f5bd7fc114c5e56d1abbfef"
server_key_iv: "3ec41c12f030693d8e9659ab84afe600"
server_key_version: "encryptionkey@0"
I20260808 13:02:01.501065 23436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance:
uuid: "8eb8ba16bceb4823a23d5214c95ccaa0"
format_stamp: "Formatted at 2026-08-08 13:02:01 on dist-test-slave-tk9z"
server_key: "fbfae1d96f5bd7fc114c5e56d1abbfef"
server_key_iv: "3ec41c12f030693d8e9659ab84afe600"
server_key_version: "encryptionkey@0"
I20260808 13:02:01.506671 23436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 13:02:01.510835 23453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:01.512247 23436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 13:02:01.512454 23436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/wal
uuid: "8eb8ba16bceb4823a23d5214c95ccaa0"
format_stamp: "Formatted at 2026-08-08 13:02:01 on dist-test-slave-tk9z"
server_key: "fbfae1d96f5bd7fc114c5e56d1abbfef"
server_key_iv: "3ec41c12f030693d8e9659ab84afe600"
server_key_version: "encryptionkey@0"
I20260808 13:02:01.512696 23436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:01.538123 23436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:01.539117 23436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:01.539480 23436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:01.540591 23436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:02:01.542191 23436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:02:01.542317 23436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:01.542490 23436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:02:01.542588 23436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:01.578672 23436 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.1:44089
I20260808 13:02:01.578771 23611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.1:44089 every 8 connection(s)
I20260808 13:02:01.580101 23436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/data/info.pb
I20260808 13:02:01.587177 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 23436
I20260808 13:02:01.587388 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-0/wal/instance
I20260808 13:02:01.587826 25244 external_mini_cluster.cc:1428] Setting key d1d0cbf34571fdd63b66747cfb8195c5
I20260808 13:02:01.591002 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.2:0
--local_ip_for_outbound_sockets=127.24.167.2
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:39575
--builtin_ntp_servers=127.24.167.20:46269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 13:02:01.595134 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:01.583064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.1:45831 (local address 127.24.167.62:39575)
0808 13:02:01.583606 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:01.583610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:01.584751 (+ 1141us) server_negotiation.cc:407] Connection header received
0808 13:02:01.587002 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:01.587008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:01.587125 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:01.587265 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:01.588605 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:01.589764 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:01.591872 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:01.592360 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:01.592522 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:01.594158 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:01.594163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:01.594167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:01.594190 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:01.594261 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:01.594282 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:01.594404 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:01.594730 (+ 326us) server_negotiation.cc:299] Negotiation successful
0808 13:02:01.594841 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":326,"threads_started":1}
I20260808 13:02:01.596685 23617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:01.583471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:39575 (local address 127.24.167.1:45831)
0808 13:02:01.584557 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:01.584607 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:01.586658 (+ 2051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:01.587582 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:01.587591 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:01.587700 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:01.588408 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:01.588428 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:01.589987 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:01.589992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:01.591685 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:01.591701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:01.592659 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:01.592670 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:01.592943 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:01.593755 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 13:02:01.593758 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:01.593813 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 13:02:01.593912 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:01.594440 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:01.594462 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:01.594475 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 13:02:01.594692 (+ 217us) client_negotiation.cc:240] Negotiation successful
0808 13:02:01.595174 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":320,"threads_started":1}
I20260808 13:02:01.598467 23612 heartbeater.cc:347] Connected to a master server at 127.24.167.62:39575
I20260808 13:02:01.598872 23612 heartbeater.cc:464] Registering TS with master...
I20260808 13:02:01.599756 23612 heartbeater.cc:511] Master 127.24.167.62:39575 requested a full tablet report, sending...
I20260808 13:02:01.602578 23349 ts_manager.cc:194] Registered new tserver with Master: 8eb8ba16bceb4823a23d5214c95ccaa0 (127.24.167.1:44089)
I20260808 13:02:01.604859 23349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.167.1:45831
W20260808 13:02:01.954290 23618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:01.954581 23618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:01.954720 23618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:01.961889 23618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:01.961979 23618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:01.962155 23618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.2
I20260808 13:02:01.971534 23618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.167.2
--webserver_port=0
--tserver_master_addrs=127.24.167.62:39575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.2
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:01.972831 23618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:01.974627 23618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:01.988075 23624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:01.988137 23625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:01.988137 23628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:01.989001 23618 server_base.cc:1034] running on GCE node
I20260808 13:02:01.989568 23618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:01.990587 23618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:01.991825 23618 hybrid_clock.cc:648] HybridClock initialized: now 1786194121991781 us; error 35 us; skew 500 ppm
I20260808 13:02:01.992208 23618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 13:02:01.994920 23618 webserver.cc:460] Webserver started at http://127.24.167.2:33761/ using document root <none> and password file <none>
I20260808 13:02:01.995692 23618 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:01.995831 23618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:01.996157 23618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:01.999336 23618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/data/instance:
uuid: "38d297c64f9d4d6bb7e99df0d8ca360a"
format_stamp: "Formatted at 2026-08-08 13:02:01 on dist-test-slave-tk9z"
server_key: "d4ecc2f1a2f916fefe0bbf2e68ccf50a"
server_key_iv: "f8073d19334c2842b7e365b32f83ca8d"
server_key_version: "encryptionkey@0"
I20260808 13:02:02.000218 23618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance:
uuid: "38d297c64f9d4d6bb7e99df0d8ca360a"
format_stamp: "Formatted at 2026-08-08 13:02:01 on dist-test-slave-tk9z"
server_key: "d4ecc2f1a2f916fefe0bbf2e68ccf50a"
server_key_iv: "f8073d19334c2842b7e365b32f83ca8d"
server_key_version: "encryptionkey@0"
I20260808 13:02:02.006170 23618 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260808 13:02:02.009888 23636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:02.011371 23618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:02:02.011584 23618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/wal
uuid: "38d297c64f9d4d6bb7e99df0d8ca360a"
format_stamp: "Formatted at 2026-08-08 13:02:01 on dist-test-slave-tk9z"
server_key: "d4ecc2f1a2f916fefe0bbf2e68ccf50a"
server_key_iv: "f8073d19334c2842b7e365b32f83ca8d"
server_key_version: "encryptionkey@0"
I20260808 13:02:02.011838 23618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:02.028789 23618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:02.029656 23618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:02.029994 23618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:02.031091 23618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:02:02.032604 23618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:02:02.032735 23618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:02.032876 23618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:02:02.032974 23618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:02.067394 23618 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.2:32977
I20260808 13:02:02.067492 23797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.2:32977 every 8 connection(s)
I20260808 13:02:02.068809 23618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/data/info.pb
I20260808 13:02:02.078294 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 23618
I20260808 13:02:02.078681 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/wal/instance
I20260808 13:02:02.079253 25244 external_mini_cluster.cc:1428] Setting key fec6e8db88d33cd4d421950442e6df20
I20260808 13:02:02.082628 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.071818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.2:46745 (local address 127.24.167.62:39575)
0808 13:02:02.072001 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:02.072010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:02.073340 (+ 1330us) server_negotiation.cc:407] Connection header received
0808 13:02:02.075232 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:02.075237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:02.075312 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:02.075408 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:02.076575 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.077332 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.079349 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.079826 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.079979 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.081703 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:02.081709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:02.081712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:02.081730 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:02.081763 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:02.081776 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:02.081849 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:02.082154 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 13:02:02.082265 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 13:02:02.082780 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.167.3:0
--local_ip_for_outbound_sockets=127.24.167.3
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:39575
--builtin_ntp_servers=127.24.167.20:46269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 13:02:02.084322 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.072238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:39575 (local address 127.24.167.2:46745)
0808 13:02:02.073172 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:02.073214 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:02.074930 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:02.075651 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:02.075660 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:02.075752 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:02.076385 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:02.076403 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.077484 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.077488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:02.079184 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:02.079198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.080097 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.080105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:02.080457 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.081296 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 13:02:02.081299 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:02.081344 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 13:02:02.081475 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:02.081892 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:02.081908 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:02.081921 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 13:02:02.082140 (+ 219us) client_negotiation.cc:240] Negotiation successful
0808 13:02:02.082657 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":281,"threads_started":1}
I20260808 13:02:02.085950 23799 heartbeater.cc:347] Connected to a master server at 127.24.167.62:39575
I20260808 13:02:02.086297 23799 heartbeater.cc:464] Registering TS with master...
I20260808 13:02:02.087174 23799 heartbeater.cc:511] Master 127.24.167.62:39575 requested a full tablet report, sending...
I20260808 13:02:02.088801 23349 ts_manager.cc:194] Registered new tserver with Master: 38d297c64f9d4d6bb7e99df0d8ca360a (127.24.167.2:32977)
I20260808 13:02:02.089658 23349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.167.2:46745
W20260808 13:02:02.433130 23803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:02.433427 23803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:02.433527 23803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:02.443049 23803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:02.443159 23803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:02.443269 23803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.167.3
I20260808 13:02:02.452412 23803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:46269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.167.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.167.3
--webserver_port=0
--tserver_master_addrs=127.24.167.62:39575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.167.3
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:02.453716 23803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:02.455523 23803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:02.468432 23808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:02.468385 23811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:02.468479 23809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:02.469563 23803 server_base.cc:1034] running on GCE node
I20260808 13:02:02.470206 23803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:02.471274 23803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:02.472515 23803 hybrid_clock.cc:648] HybridClock initialized: now 1786194122472457 us; error 75 us; skew 500 ppm
I20260808 13:02:02.472846 23803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 13:02:02.475358 23803 webserver.cc:460] Webserver started at http://127.24.167.3:36733/ using document root <none> and password file <none>
I20260808 13:02:02.476068 23803 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:02.476208 23803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:02.476541 23803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:02.479549 23803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/data/instance:
uuid: "ba4435283a074232a8cc5fb9df16a3a4"
format_stamp: "Formatted at 2026-08-08 13:02:02 on dist-test-slave-tk9z"
server_key: "d7241196003d083a12b5e18a2596d529"
server_key_iv: "d68c691e9934f8563c06e2579797f2c1"
server_key_version: "encryptionkey@0"
I20260808 13:02:02.480381 23803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance:
uuid: "ba4435283a074232a8cc5fb9df16a3a4"
format_stamp: "Formatted at 2026-08-08 13:02:02 on dist-test-slave-tk9z"
server_key: "d7241196003d083a12b5e18a2596d529"
server_key_iv: "d68c691e9934f8563c06e2579797f2c1"
server_key_version: "encryptionkey@0"
I20260808 13:02:02.485795 23803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260808 13:02:02.489398 23819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:02.490824 23803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:02:02.491010 23803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/wal
uuid: "ba4435283a074232a8cc5fb9df16a3a4"
format_stamp: "Formatted at 2026-08-08 13:02:02 on dist-test-slave-tk9z"
server_key: "d7241196003d083a12b5e18a2596d529"
server_key_iv: "d68c691e9934f8563c06e2579797f2c1"
server_key_version: "encryptionkey@0"
I20260808 13:02:02.491240 23803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:02.510687 23803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:02.511530 23803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:02.511852 23803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:02.512862 23803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 13:02:02.514508 23803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 13:02:02.514657 23803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:02.514789 23803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 13:02:02.514855 23803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:02.547605 23803 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.3:43201
I20260808 13:02:02.547654 23978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.3:43201 every 8 connection(s)
I20260808 13:02:02.549017 23803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/data/info.pb
I20260808 13:02:02.557106 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 23803
I20260808 13:02:02.557292 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/wal/instance
I20260808 13:02:02.557797 25244 external_mini_cluster.cc:1428] Setting key fd0e3bbc2a172210389fcba00fbcff03
I20260808 13:02:02.561810 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.551711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:42649 (local address 127.24.167.62:39575)
0808 13:02:02.551900 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:02.551905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:02.553316 (+ 1411us) server_negotiation.cc:407] Connection header received
0808 13:02:02.555109 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:02.555113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:02.555185 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:02.555274 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:02.556292 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.557361 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.559051 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.559465 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.559607 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.560988 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:02.560994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:02.560997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:02.561014 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:02.561047 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:02.561059 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:02.561157 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:02.561373 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 13:02:02.561478 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 13:02:02.562865 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.552272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:39575 (local address 127.24.167.3:42649)
0808 13:02:02.553160 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:02.553201 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:02.554871 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:02.555509 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:02.555516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:02.555596 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:02.556140 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:02.556155 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.557531 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.557535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:02.558898 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:02.558910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.559693 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.559701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:02.559947 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.560660 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 13:02:02.560662 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:02.560700 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 13:02:02.560779 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:02.561155 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:02.561168 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:02.561178 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 13:02:02.561361 (+ 183us) client_negotiation.cc:240] Negotiation successful
0808 13:02:02.561775 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":289,"threads_started":1}
I20260808 13:02:02.564179 23979 heartbeater.cc:347] Connected to a master server at 127.24.167.62:39575
I20260808 13:02:02.564483 23979 heartbeater.cc:464] Registering TS with master...
I20260808 13:02:02.565251 23979 heartbeater.cc:511] Master 127.24.167.62:39575 requested a full tablet report, sending...
I20260808 13:02:02.566836 23349 ts_manager.cc:194] Registered new tserver with Master: ba4435283a074232a8cc5fb9df16a3a4 (127.24.167.3:43201)
I20260808 13:02:02.567634 23349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.167.3:42649
I20260808 13:02:02.573169 25244 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 13:02:02.584262 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.577230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.24.167.62:39575)
0808 13:02:02.577458 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:02.577468 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:02.577875 (+ 407us) server_negotiation.cc:407] Connection header received
0808 13:02:02.578095 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:02.578100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:02.578182 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:02.578265 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:02.578911 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.579663 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.581621 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.582061 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.582161 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.583550 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:02.583555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:02.583558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:02.583577 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:02.583611 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:02.583623 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:02.583695 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:02.583872 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 13:02:02.583950 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 13:02:02.588310 23349 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 13:02:02.595443 23349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 13:02:02.609452 23612 heartbeater.cc:503] Master 127.24.167.62:39575 was elected leader, sending a full tablet report...
I20260808 13:02:02.620244 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.607694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:44089 (local address 127.0.0.1:52920)
0808 13:02:02.608503 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:02.608540 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:02.608712 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:02.612866 (+ 4154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:02.612870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:02.612894 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:02:02.613337 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:02.613350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.615297 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.615303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:02.617638 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:02.617649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.619116 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.619121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:02.619388 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.619484 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 13:02:02.619604 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 13:02:02.619797 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":75,"thread_start_us":396,"threads_started":1}
I20260808 13:02:02.620265 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.610398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.24.167.2:32977)
0808 13:02:02.611099 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:02.611107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:02.611655 (+ 548us) server_negotiation.cc:407] Connection header received
0808 13:02:02.611960 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:02.611967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:02.612199 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:02.612367 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:02:02.613324 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.614436 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.617414 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.618835 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.618977 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.619231 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:02.619707 (+ 476us) server_negotiation.cc:299] Negotiation successful
0808 13:02:02.619884 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":403,"threads_started":1}
I20260808 13:02:02.620265 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.607895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.24.167.1:44089)
0808 13:02:02.612322 (+ 4427us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:02.612332 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:02.612359 (+ 27us) server_negotiation.cc:407] Connection header received
0808 13:02:02.612446 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:02.612453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:02.612671 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:02.612867 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:02:02.613537 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.615101 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.617864 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.618879 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.619372 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.619659 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:02.619723 (+ 64us) server_negotiation.cc:299] Negotiation successful
0808 13:02:02.619884 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2560,"thread_start_us":2384,"threads_started":1}
I20260808 13:02:02.621018 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.610834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:32977 (local address 127.0.0.1:41340)
0808 13:02:02.611544 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:02.611561 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:02.611760 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:02.612646 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:02.612653 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:02.612681 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:02:02.613128 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:02.613141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.614585 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.614591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:02.617148 (+ 2557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:02.617164 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.619044 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.619051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:02.619334 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.619477 (+ 143us) client_negotiation.cc:769] Sending connection context
0808 13:02:02.620488 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0808 13:02:02.620646 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":104,"thread_start_us":415,"threads_started":1}
I20260808 13:02:02.621627 23999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.608387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.24.167.3:43201)
0808 13:02:02.610416 (+ 2029us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:02.610424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:02.610651 (+ 227us) server_negotiation.cc:407] Connection header received
0808 13:02:02.611017 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:02.611025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:02.611306 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:02.611529 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:02:02.612559 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.614189 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.617272 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.618835 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.618972 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.619261 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:02.621127 (+ 1866us) server_negotiation.cc:299] Negotiation successful
0808 13:02:02.621302 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1801,"thread_start_us":1650,"threads_started":1}
I20260808 13:02:02.621675 23996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.607821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.3:43201 (local address 127.0.0.1:33774)
0808 13:02:02.610512 (+ 2691us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:02.610541 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:02.610786 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:02.611843 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:02.611853 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:02.611884 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:02:02.612303 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:02.612321 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.614336 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.614341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:02.617045 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:02.617058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.620484 (+ 3426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.620498 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:02.620679 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.620709 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 13:02:02.620801 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 13:02:02.620946 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1247,"mutex_wait_us":91,"thread_start_us":1346,"threads_started":1}
I20260808 13:02:02.625082 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 2476d8c19f4a426eb69466fb11b74b0a (DEFAULT_TABLE table=test-table [id=b44274af415546beb7e57d27edda7d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:02:02.625052 23519 tablet_service.cc:1467] Processing CreateTablet for tablet 2476d8c19f4a426eb69466fb11b74b0a (DEFAULT_TABLE table=test-table [id=b44274af415546beb7e57d27edda7d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:02:02.625051 23700 tablet_service.cc:1467] Processing CreateTablet for tablet 2476d8c19f4a426eb69466fb11b74b0a (DEFAULT_TABLE table=test-table [id=b44274af415546beb7e57d27edda7d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 13:02:02.626585 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2476d8c19f4a426eb69466fb11b74b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:02.626616 23519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2476d8c19f4a426eb69466fb11b74b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:02.626613 23700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2476d8c19f4a426eb69466fb11b74b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:02.639178 24002 tablet_bootstrap.cc:492] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4: Bootstrap starting.
I20260808 13:02:02.641275 24003 tablet_bootstrap.cc:492] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0: Bootstrap starting.
I20260808 13:02:02.642108 24004 tablet_bootstrap.cc:492] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a: Bootstrap starting.
I20260808 13:02:02.642743 24002 tablet_bootstrap.cc:654] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:02.644083 24002 log.cc:824] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:02.646078 24004 tablet_bootstrap.cc:654] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:02.647172 24002 tablet_bootstrap.cc:492] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4: No bootstrap required, opened a new log
I20260808 13:02:02.647500 24004 log.cc:824] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:02.647612 24002 ts_tablet_manager.cc:1397] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260808 13:02:02.648475 24003 tablet_bootstrap.cc:654] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:02.649725 24003 log.cc:824] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:02.650830 24004 tablet_bootstrap.cc:492] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a: No bootstrap required, opened a new log
I20260808 13:02:02.651155 24004 ts_tablet_manager.cc:1397] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 13:02:02.652966 24003 tablet_bootstrap.cc:492] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0: No bootstrap required, opened a new log
I20260808 13:02:02.653244 24003 ts_tablet_manager.cc:1397] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20260808 13:02:02.655575 24002 raft_consensus.cc:348] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb8ba16bceb4823a23d5214c95ccaa0" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44089 } } peers { permanent_uuid: "ba4435283a074232a8cc5fb9df16a3a4" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43201 } } peers { permanent_uuid: "38d297c64f9d4d6bb7e99df0d8ca360a" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 32977 } }
I20260808 13:02:02.655966 24002 raft_consensus.cc:374] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:02.656117 24002 raft_consensus.cc:729] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4435283a074232a8cc5fb9df16a3a4, State: Initialized, Role: FOLLOWER
I20260808 13:02:02.656735 24002 consensus_queue.cc:260] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb8ba16bceb4823a23d5214c95ccaa0" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44089 } } peers { permanent_uuid: "ba4435283a074232a8cc5fb9df16a3a4" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43201 } } peers { permanent_uuid: "38d297c64f9d4d6bb7e99df0d8ca360a" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 32977 } }
I20260808 13:02:02.659621 23979 heartbeater.cc:503] Master 127.24.167.62:39575 was elected leader, sending a full tablet report...
I20260808 13:02:02.661594 24002 ts_tablet_manager.cc:1428] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4: Time spent starting tablet: real 0.014s user 0.006s sys 0.006s
I20260808 13:02:02.661907 24004 raft_consensus.cc:348] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb8ba16bceb4823a23d5214c95ccaa0" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44089 } } peers { permanent_uuid: "ba4435283a074232a8cc5fb9df16a3a4" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43201 } } peers { permanent_uuid: "38d297c64f9d4d6bb7e99df0d8ca360a" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 32977 } }
I20260808 13:02:02.662529 24004 raft_consensus.cc:374] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:02.662670 24004 raft_consensus.cc:729] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d297c64f9d4d6bb7e99df0d8ca360a, State: Initialized, Role: FOLLOWER
I20260808 13:02:02.663421 24004 consensus_queue.cc:260] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb8ba16bceb4823a23d5214c95ccaa0" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44089 } } peers { permanent_uuid: "ba4435283a074232a8cc5fb9df16a3a4" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43201 } } peers { permanent_uuid: "38d297c64f9d4d6bb7e99df0d8ca360a" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 32977 } }
I20260808 13:02:02.663910 24003 raft_consensus.cc:348] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb8ba16bceb4823a23d5214c95ccaa0" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44089 } } peers { permanent_uuid: "ba4435283a074232a8cc5fb9df16a3a4" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43201 } } peers { permanent_uuid: "38d297c64f9d4d6bb7e99df0d8ca360a" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 32977 } }
I20260808 13:02:02.664438 24003 raft_consensus.cc:374] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:02.664587 24003 raft_consensus.cc:729] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb8ba16bceb4823a23d5214c95ccaa0, State: Initialized, Role: FOLLOWER
I20260808 13:02:02.664605 23799 heartbeater.cc:503] Master 127.24.167.62:39575 was elected leader, sending a full tablet report...
I20260808 13:02:02.665333 24004 ts_tablet_manager.cc:1428] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260808 13:02:02.665369 24003 consensus_queue.cc:260] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb8ba16bceb4823a23d5214c95ccaa0" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44089 } } peers { permanent_uuid: "ba4435283a074232a8cc5fb9df16a3a4" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43201 } } peers { permanent_uuid: "38d297c64f9d4d6bb7e99df0d8ca360a" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 32977 } }
I20260808 13:02:02.667413 24003 ts_tablet_manager.cc:1428] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
W20260808 13:02:02.800832 23980 tablet.cc:2369] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 13:02:02.821082 23801 tablet.cc:2369] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 13:02:02.832885 23614 tablet.cc:2369] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:02:02.884332 24009 raft_consensus.cc:482] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 13:02:02.884689 24009 raft_consensus.cc:504] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb8ba16bceb4823a23d5214c95ccaa0" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44089 } } peers { permanent_uuid: "ba4435283a074232a8cc5fb9df16a3a4" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43201 } } peers { permanent_uuid: "38d297c64f9d4d6bb7e99df0d8ca360a" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 32977 } }
I20260808 13:02:02.886680 24009 leader_election.cc:288] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb8ba16bceb4823a23d5214c95ccaa0 (127.24.167.1:44089), 38d297c64f9d4d6bb7e99df0d8ca360a (127.24.167.2:32977)
I20260808 13:02:02.893540 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.887255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:44089 (local address 127.24.167.3:50447)
0808 13:02:02.887430 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:02.887481 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:02.887616 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:02.888014 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:02.888018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:02.888051 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:02:02.888262 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:02.888277 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.889596 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.889600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:02.891579 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:02.891595 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.892970 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.892974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:02.893094 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.893119 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 13:02:02.893194 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 13:02:02.893295 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":1}
I20260808 13:02:02.893648 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.887301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:50447 (local address 127.24.167.1:44089)
0808 13:02:02.887534 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:02.887541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:02.887568 (+ 27us) server_negotiation.cc:407] Connection header received
0808 13:02:02.887752 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:02.887757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:02.887835 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:02.887965 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:02:02.888469 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.889459 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.891756 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.892810 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.892974 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.893048 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:02.893254 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 13:02:02.893390 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 13:02:02.894547 24014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.887255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.2:32977 (local address 127.24.167.3:44545)
0808 13:02:02.888461 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:02.888475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:02.888578 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:02.888904 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:02.888907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:02.888926 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 13:02:02.889229 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:02.889236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.890318 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.890323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:02.892133 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:02.892143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.893717 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.893726 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:02.893929 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.893981 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 13:02:02.894117 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 13:02:02.894208 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074,"mutex_wait_us":436,"thread_start_us":442,"threads_started":1}
I20260808 13:02:02.894675 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.887309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.167.3:44545 (local address 127.24.167.2:32977)
0808 13:02:02.887535 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:02.887541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:02.888583 (+ 1042us) server_negotiation.cc:407] Connection header received
0808 13:02:02.888675 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:02.888680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:02.888789 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:02.888882 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 13:02:02.889403 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.890142 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.892333 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.893529 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.893690 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.893764 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:02.894184 (+ 420us) server_negotiation.cc:299] Negotiation successful
0808 13:02:02.894329 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 13:02:02.895103 23546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2476d8c19f4a426eb69466fb11b74b0a" candidate_uuid: "ba4435283a074232a8cc5fb9df16a3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb8ba16bceb4823a23d5214c95ccaa0" is_pre_election: true
I20260808 13:02:02.895749 23546 raft_consensus.cc:2454] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba4435283a074232a8cc5fb9df16a3a4 in term 0.
I20260808 13:02:02.895790 23730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2476d8c19f4a426eb69466fb11b74b0a" candidate_uuid: "ba4435283a074232a8cc5fb9df16a3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d297c64f9d4d6bb7e99df0d8ca360a" is_pre_election: true
I20260808 13:02:02.896466 23730 raft_consensus.cc:2454] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba4435283a074232a8cc5fb9df16a3a4 in term 0.
I20260808 13:02:02.896613 23821 leader_election.cc:302] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8eb8ba16bceb4823a23d5214c95ccaa0, ba4435283a074232a8cc5fb9df16a3a4; no voters:
I20260808 13:02:02.897073 24009 raft_consensus.cc:2781] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 13:02:02.897253 24009 raft_consensus.cc:482] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 13:02:02.897367 24009 raft_consensus.cc:3037] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:02.900939 24009 raft_consensus.cc:504] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb8ba16bceb4823a23d5214c95ccaa0" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44089 } } peers { permanent_uuid: "ba4435283a074232a8cc5fb9df16a3a4" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43201 } } peers { permanent_uuid: "38d297c64f9d4d6bb7e99df0d8ca360a" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 32977 } }
I20260808 13:02:02.902432 24009 leader_election.cc:288] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [CANDIDATE]: Term 1 election: Requested vote from peers 8eb8ba16bceb4823a23d5214c95ccaa0 (127.24.167.1:44089), 38d297c64f9d4d6bb7e99df0d8ca360a (127.24.167.2:32977)
I20260808 13:02:02.902575 23546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2476d8c19f4a426eb69466fb11b74b0a" candidate_uuid: "ba4435283a074232a8cc5fb9df16a3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb8ba16bceb4823a23d5214c95ccaa0"
I20260808 13:02:02.902575 23730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2476d8c19f4a426eb69466fb11b74b0a" candidate_uuid: "ba4435283a074232a8cc5fb9df16a3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d297c64f9d4d6bb7e99df0d8ca360a"
I20260808 13:02:02.902812 23730 raft_consensus.cc:3037] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:02.902812 23546 raft_consensus.cc:3037] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:02.906158 23546 raft_consensus.cc:2454] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba4435283a074232a8cc5fb9df16a3a4 in term 1.
I20260808 13:02:02.906164 23730 raft_consensus.cc:2454] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba4435283a074232a8cc5fb9df16a3a4 in term 1.
I20260808 13:02:02.906679 23820 leader_election.cc:302] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d297c64f9d4d6bb7e99df0d8ca360a, ba4435283a074232a8cc5fb9df16a3a4; no voters:
I20260808 13:02:02.908186 24009 raft_consensus.cc:2781] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:02.908748 24009 raft_consensus.cc:686] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [term 1 LEADER]: Becoming Leader. State: Replica: ba4435283a074232a8cc5fb9df16a3a4, State: Running, Role: LEADER
I20260808 13:02:02.909403 24009 consensus_queue.cc:237] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb8ba16bceb4823a23d5214c95ccaa0" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44089 } } peers { permanent_uuid: "ba4435283a074232a8cc5fb9df16a3a4" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43201 } } peers { permanent_uuid: "38d297c64f9d4d6bb7e99df0d8ca360a" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 32977 } }
I20260808 13:02:02.915503 23348 catalog_manager.cc:5495] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba4435283a074232a8cc5fb9df16a3a4 (127.24.167.3). New cstate: current_term: 1 leader_uuid: "ba4435283a074232a8cc5fb9df16a3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb8ba16bceb4823a23d5214c95ccaa0" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba4435283a074232a8cc5fb9df16a3a4" member_type: VOTER last_known_addr { host: "127.24.167.3" port: 43201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38d297c64f9d4d6bb7e99df0d8ca360a" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 32977 } health_report { overall_health: UNKNOWN } } }
I20260808 13:02:02.991158 23999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:02.985375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.24.167.3:43201)
0808 13:02:02.985564 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:02.985568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:02.985589 (+ 21us) server_negotiation.cc:407] Connection header received
0808 13:02:02.985744 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:02.985748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:02.985841 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:02.985997 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:02.986326 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.987157 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.988475 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:02.988935 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:02.989103 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:02.989852 (+ 749us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:02.990684 (+ 832us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:02.990797 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:02.990900 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 13:02:02.990965 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 13:02:02.999956 23730 raft_consensus.cc:1261] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a [term 1 FOLLOWER]: Refusing update from remote peer ba4435283a074232a8cc5fb9df16a3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:02:02.999948 23546 raft_consensus.cc:1261] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0 [term 1 FOLLOWER]: Refusing update from remote peer ba4435283a074232a8cc5fb9df16a3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:02:03.001312 24009 consensus_queue.cc:1035] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38d297c64f9d4d6bb7e99df0d8ca360a" member_type: VOTER last_known_addr { host: "127.24.167.2" port: 32977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:02:03.001726 24015 consensus_queue.cc:1035] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb8ba16bceb4823a23d5214c95ccaa0" member_type: VOTER last_known_addr { host: "127.24.167.1" port: 44089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:02:03.028460 23348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:02:03.028771 23348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 13:02:03.032881 23348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1975ba4d43f34bdd967d9a5108e658e4: Sending DeleteTablet for 3 replicas of tablet 2476d8c19f4a426eb69466fb11b74b0a
I20260808 13:02:03.034097 23519 tablet_service.cc:1514] Processing DeleteTablet for tablet 2476d8c19f4a426eb69466fb11b74b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:02:03 UTC) from {username='slave'} at 127.0.0.1:52920
I20260808 13:02:03.034107 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet 2476d8c19f4a426eb69466fb11b74b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:02:03 UTC) from {username='slave'} at 127.0.0.1:33774
I20260808 13:02:03.034235 23700 tablet_service.cc:1514] Processing DeleteTablet for tablet 2476d8c19f4a426eb69466fb11b74b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 13:02:03 UTC) from {username='slave'} at 127.0.0.1:41340
I20260808 13:02:03.037178 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 23436
I20260808 13:02:03.046986 24034 tablet_replica.cc:330] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a: stopping tablet replica
I20260808 13:02:03.048101 24034 raft_consensus.cc:2229] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:02:03.049544 24034 raft_consensus.cc:2258] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:02:03.050462 24033 tablet_replica.cc:330] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0: stopping tablet replica
I20260808 13:02:03.051430 24033 raft_consensus.cc:2229] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 13:02:03.054214 24034 ts_tablet_manager.cc:1905] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:02:03.054777 24033 raft_consensus.cc:2258] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:02:03.057047 24033 ts_tablet_manager.cc:1905] T 2476d8c19f4a426eb69466fb11b74b0a P 8eb8ba16bceb4823a23d5214c95ccaa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:02:03.060091 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:03.038859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.24.167.1:44089)
0808 13:02:03.039056 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:03.039064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:03.051574 (+ 12510us) server_negotiation.cc:407] Connection header received
0808 13:02:03.052056 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:03.052062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:03.052161 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:03.052264 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:03.052993 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:03.054135 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:03.056016 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:03.056675 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:03.056772 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:03.058262 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:03.058273 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:03.058291 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:03.059076 (+ 785us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:03.059141 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:03.059170 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:03.059274 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:03.059576 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 13:02:03.059695 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 13:02:03.061298 23606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:03.062444 24032 tablet_replica.cc:330] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4: stopping tablet replica
I20260808 13:02:03.062891 24032 raft_consensus.cc:2229] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [term 1 LEADER]: Raft consensus shutting down.
I20260808 13:02:03.063638 24032 raft_consensus.cc:2258] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 13:02:03.063735 24034 ts_tablet_manager.cc:1918] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:02:03.064083 24034 log.cc:1196] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-1/wal/wals/2476d8c19f4a426eb69466fb11b74b0a
I20260808 13:02:03.064834 24034 ts_tablet_manager.cc:1939] T 2476d8c19f4a426eb69466fb11b74b0a P 38d297c64f9d4d6bb7e99df0d8ca360a: Deleting consensus metadata
I20260808 13:02:03.067147 23329 catalog_manager.cc:4841] TS 38d297c64f9d4d6bb7e99df0d8ca360a (127.24.167.2:32977): tablet 2476d8c19f4a426eb69466fb11b74b0a (table test-table [id=b44274af415546beb7e57d27edda7d94]) successfully deleted
I20260808 13:02:03.068339 24032 ts_tablet_manager.cc:1905] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 13:02:03.074735 24032 ts_tablet_manager.cc:1918] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 13:02:03.074925 24032 log.cc:1196] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4: Deleting WAL directory at /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786193784050718-25244-0/minicluster-data/ts-2/wal/wals/2476d8c19f4a426eb69466fb11b74b0a
I20260808 13:02:03.075503 24032 ts_tablet_manager.cc:1939] T 2476d8c19f4a426eb69466fb11b74b0a P ba4435283a074232a8cc5fb9df16a3a4: Deleting consensus metadata
I20260808 13:02:03.077416 23332 catalog_manager.cc:4841] TS ba4435283a074232a8cc5fb9df16a3a4 (127.24.167.3:43201): tablet 2476d8c19f4a426eb69466fb11b74b0a (table test-table [id=b44274af415546beb7e57d27edda7d94]) successfully deleted
I20260808 13:02:03.163270 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 23436
W20260808 13:02:03.196141 23330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.167.1:44089 (error 108)
I20260808 13:02:03.196344 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 23618
I20260808 13:02:03.198520 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:03.197694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:44089 (local address 127.0.0.1:52942)
0808 13:02:03.198172 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:03.198278 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":244,"threads_started":1}
W20260808 13:02:03.198741 23330 catalog_manager.cc:4584] TS 8eb8ba16bceb4823a23d5214c95ccaa0 (127.24.167.1:44089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2476d8c19f4a426eb69466fb11b74b0a: Network error: Client connection negotiation failed: client connection to 127.24.167.1:44089: connect: Connection refused (error 111)
I20260808 13:02:03.205345 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:03.197218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.24.167.2:32977)
0808 13:02:03.197419 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:03.197425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:03.197509 (+ 84us) server_negotiation.cc:407] Connection header received
0808 13:02:03.197717 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:03.197724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:03.197835 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:03.197958 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:03.198697 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:03.199886 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:03.201398 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:03.202133 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:03.202266 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:03.203680 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:03.203691 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:03.203711 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:03.204517 (+ 806us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:03.204582 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:03.204609 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:03.204738 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:03.204839 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 13:02:03.204956 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 13:02:03.206161 23793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:03.252794 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:03.252209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:44089 (local address 127.0.0.1:52946)
0808 13:02:03.252513 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:03.252614 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260808 13:02:03.253990 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:03.253556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:44089 (local address 127.0.0.1:52956)
0808 13:02:03.253759 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:03.253832 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 13:02:03.296640 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 23618
I20260808 13:02:03.314823 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:03.314214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:44089 (local address 127.0.0.1:52966)
0808 13:02:03.314504 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:03.314623 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 13:02:03.316200 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:03.315691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:44089 (local address 127.0.0.1:52968)
0808 13:02:03.315884 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:03.315997 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 13:02:03.326639 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 23803
I20260808 13:02:03.334965 23999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:03.327673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33804 (local address 127.24.167.3:43201)
0808 13:02:03.327859 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:03.327864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:03.327882 (+ 18us) server_negotiation.cc:407] Connection header received
0808 13:02:03.328001 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:03.328005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:03.328103 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:03.328190 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:03.328690 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:03.329479 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:03.331415 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:03.331878 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:03.331986 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:03.333380 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:03.333390 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:03.333406 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:03.334050 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:03.334103 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:03.334125 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:03.334222 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:03.334482 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 13:02:03.334573 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 13:02:03.335764 23974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:03.396869 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:03.396377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:44089 (local address 127.0.0.1:52978)
0808 13:02:03.396610 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:03.396716 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260808 13:02:03.398027 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:03.397587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:44089 (local address 127.0.0.1:52986)
0808 13:02:03.397796 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:03.397866 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260808 13:02:03.460044 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 23803
I20260808 13:02:03.488090 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 23306
I20260808 13:02:03.489337 23392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:03.603659 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:03.602982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.1:44089 (local address 127.0.0.1:52998)
0808 13:02:03.603286 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:03.603419 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.1:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 13:02:03.603729 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 23306
2026-08-08T13:02:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3231 ms)
[----------] 6 tests from EncryptionPolicyTest (24013 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:03 dist-test-slave-tk9z krb5kdc[24048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:03 dist-test-slave-tk9z krb5kdc[24048](info): set up 2 sockets
Aug 08 13:02:03 dist-test-slave-tk9z krb5kdc[24048](info): commencing operation
krb5kdc: starting...
W20260808 13:02:04.948560 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.292s user 0.001s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:05 dist-test-slave-tk9z krb5kdc[24048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 13:02:05.274072 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:05.630753 24068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:05.631068 24068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:05.631165 24068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:05.638550 24068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:05.638669 24068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:05.638716 24068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:05.638779 24068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:05.638830 24068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:05.649055 24068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:05.650472 24068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:05.651773 24068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:05.664177 24077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:05.664304 24079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:05.664354 24083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:05.665149 24068 server_base.cc:1034] running on GCE node
I20260808 13:02:05.666033 24068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:05.667588 24068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:05.668812 24068 hybrid_clock.cc:648] HybridClock initialized: now 1786194125668753 us; error 40 us; skew 500 ppm
I20260808 13:02:05.669204 24068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:05 dist-test-slave-tk9z krb5kdc[24048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:05.672587 24068 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 13:02:05.674160 24068 webserver.cc:460] Webserver started at http://127.0.0.1:41387/ using document root <none> and password file <none>
I20260808 13:02:05.674933 24068 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:05.675073 24068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:05.675423 24068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:05.678766 24068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "12be3188cc0c4f6e9132641a2f1a91b8"
format_stamp: "Formatted at 2026-08-08 13:02:05 on dist-test-slave-tk9z"
server_key: "7d8da1d3c609e89ebf46532a5e2ed588"
server_key_iv: "c6f5a172b354008f858a8aaec28111bb"
server_key_version: "encryptionkey@0"
I20260808 13:02:05.679629 24068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "12be3188cc0c4f6e9132641a2f1a91b8"
format_stamp: "Formatted at 2026-08-08 13:02:05 on dist-test-slave-tk9z"
server_key: "7d8da1d3c609e89ebf46532a5e2ed588"
server_key_iv: "c6f5a172b354008f858a8aaec28111bb"
server_key_version: "encryptionkey@0"
I20260808 13:02:05.685473 24068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 13:02:05.689155 24091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:05.690685 24068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 13:02:05.690879 24068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "12be3188cc0c4f6e9132641a2f1a91b8"
format_stamp: "Formatted at 2026-08-08 13:02:05 on dist-test-slave-tk9z"
server_key: "7d8da1d3c609e89ebf46532a5e2ed588"
server_key_iv: "c6f5a172b354008f858a8aaec28111bb"
server_key_version: "encryptionkey@0"
I20260808 13:02:05.691080 24068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:05.705552 24068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:05.710314 24068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:05.710691 24068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:05.730360 24068 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38985
I20260808 13:02:05.730444 24156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38985 every 8 connection(s)
I20260808 13:02:05.732172 24068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:05.735911 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 24068
I20260808 13:02:05.736163 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:05.736326 24157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:05.736869 25244 external_mini_cluster.cc:1428] Setting key 57a78bf9ec23c2b4956c79007404ffa2
I20260808 13:02:05.751801 24157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8: Bootstrap starting.
Aug 08 13:02:05 dist-test-slave-tk9z krb5kdc[24048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 13:02:05.755247 24157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:05.756479 24157 log.cc:824] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:05.759514 24157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8: No bootstrap required, opened a new log
I20260808 13:02:05.761796 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:05.738851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46331 (local address 127.0.0.1:38985)
0808 13:02:05.739842 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:05.739854 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:05.739895 (+ 41us) server_negotiation.cc:407] Connection header received
0808 13:02:05.741484 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:05.741507 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:05.741897 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:05.742223 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:05.744183 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:05.745469 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:05.747343 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:05.748086 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:05.748240 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:05.753741 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:05.753749 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:05.753764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:05.753810 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:05.757086 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:05.757914 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:05.757921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:05.757927 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:05.758096 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:05.758690 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:05.758694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:05.758696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:05.759189 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:05.759544 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:05.760058 (+ 514us) server_negotiation.cc:299] Negotiation successful
0808 13:02:05.760445 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":516,"threads_started":1}
I20260808 13:02:05.765961 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:05.768321 24157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12be3188cc0c4f6e9132641a2f1a91b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38985 } }
I20260808 13:02:05.768705 24157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:05.768843 24157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12be3188cc0c4f6e9132641a2f1a91b8, State: Initialized, Role: FOLLOWER
I20260808 13:02:05.769457 24157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12be3188cc0c4f6e9132641a2f1a91b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38985 } }
I20260808 13:02:05.769723 24157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:05.769881 24157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:05.770053 24157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:05.772889 24157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12be3188cc0c4f6e9132641a2f1a91b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38985 } }
I20260808 13:02:05.773418 24157 leader_election.cc:302] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12be3188cc0c4f6e9132641a2f1a91b8; no voters:
I20260808 13:02:05.774170 24157 leader_election.cc:288] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:05.774312 24171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:05.776664 24171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [term 1 LEADER]: Becoming Leader. State: Replica: 12be3188cc0c4f6e9132641a2f1a91b8, State: Running, Role: LEADER
I20260808 13:02:05.776902 24157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:05.777381 24171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12be3188cc0c4f6e9132641a2f1a91b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38985 } }
I20260808 13:02:05.781335 24172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12be3188cc0c4f6e9132641a2f1a91b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12be3188cc0c4f6e9132641a2f1a91b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38985 } } }
I20260808 13:02:05.781754 24172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:05.782600 24174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12be3188cc0c4f6e9132641a2f1a91b8. Latest consensus state: current_term: 1 leader_uuid: "12be3188cc0c4f6e9132641a2f1a91b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12be3188cc0c4f6e9132641a2f1a91b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38985 } } }
I20260808 13:02:05.782982 24174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:05.792440 24183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:05.792801 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:05.771159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55819 (local address 127.0.0.1:38985)
0808 13:02:05.771364 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:05.771370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:05.771813 (+ 443us) server_negotiation.cc:407] Connection header received
0808 13:02:05.771954 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:05.771959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:05.772046 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:05.772164 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:05.773748 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:05.774867 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:05.776324 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:05.777069 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:05.777216 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:05.781567 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:05.781578 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:05.781584 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:05.781629 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:05.786925 (+ 5296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:05.788367 (+ 1442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:05.788376 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:05.788382 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:05.788579 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:05.789320 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:05.789328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:05.789333 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:05.789894 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:05.790318 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:05.791829 (+ 1511us) server_negotiation.cc:299] Negotiation successful
0808 13:02:05.792098 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 13:02:05.798776 24183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:05.807143 24183 catalog_manager.cc:1347] Generated new cluster ID: 2d3aad07505b4222a3ea246aa82562bb
I20260808 13:02:05.807262 24183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:05.839892 24183 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:05.841840 24183 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:05.852520 24183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12be3188cc0c4f6e9132641a2f1a91b8: Generated new TSK 0
I20260808 13:02:05.853523 24183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:05.859208 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 24068
I20260808 13:02:05.860424 24150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:05.955986 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 24068
W20260808 13:02:05.983349 24046 connection.cc:535] client connection to 127.0.0.1:38985 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38985): Connection reset by peer (error 104)
2026-08-08T13:02:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:06 dist-test-slave-tk9z krb5kdc[24204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:06 dist-test-slave-tk9z krb5kdc[24204](info): set up 2 sockets
Aug 08 13:02:06 dist-test-slave-tk9z krb5kdc[24204](info): commencing operation
krb5kdc: starting...
W20260808 13:02:07.294883 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.292s user 0.002s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:07 dist-test-slave-tk9z krb5kdc[24204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 13:02:07.617996 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:07.985016 24220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:07.985448 24220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:07.985618 24220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:07.993177 24220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:07.993347 24220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:07.993464 24220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:07.993585 24220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:07.993665 24220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:08.004158 24220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44295
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:08.005623 24220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:08.007035 24220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:08.019755 24228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:08.019768 24229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:08.019767 24231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:08.020857 24220 server_base.cc:1034] running on GCE node
I20260808 13:02:08.021620 24220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:08.023146 24220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:08.024400 24220 hybrid_clock.cc:648] HybridClock initialized: now 1786194128024322 us; error 29 us; skew 500 ppm
I20260808 13:02:08.024732 24220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:08 dist-test-slave-tk9z krb5kdc[24204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:08.028115 24220 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 13:02:08.029642 24220 webserver.cc:460] Webserver started at http://127.0.0.1:41689/ using document root <none> and password file <none>
I20260808 13:02:08.030324 24220 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:08.030504 24220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:08.030783 24220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:08.034158 24220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "56e89e722a1946dfb01f93663eacda43"
format_stamp: "Formatted at 2026-08-08 13:02:08 on dist-test-slave-tk9z"
server_key: "1367f1866a43b3515beab2b3669bb25a"
server_key_iv: "77fe7a10152c46a69e6aaf8df4b5c0fc"
server_key_version: "encryptionkey@0"
I20260808 13:02:08.035046 24220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "56e89e722a1946dfb01f93663eacda43"
format_stamp: "Formatted at 2026-08-08 13:02:08 on dist-test-slave-tk9z"
server_key: "1367f1866a43b3515beab2b3669bb25a"
server_key_iv: "77fe7a10152c46a69e6aaf8df4b5c0fc"
server_key_version: "encryptionkey@0"
I20260808 13:02:08.040300 24220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 13:02:08.043849 24239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:08.045166 24220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 13:02:08.045306 24220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "56e89e722a1946dfb01f93663eacda43"
format_stamp: "Formatted at 2026-08-08 13:02:08 on dist-test-slave-tk9z"
server_key: "1367f1866a43b3515beab2b3669bb25a"
server_key_iv: "77fe7a10152c46a69e6aaf8df4b5c0fc"
server_key_version: "encryptionkey@0"
I20260808 13:02:08.045499 24220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:08.065225 24220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:08.069881 24220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:08.070133 24220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:08.089180 24220 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44295
I20260808 13:02:08.089226 24308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44295 every 8 connection(s)
I20260808 13:02:08.090642 24220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:08.094558 24309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:08.100508 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 24220
I20260808 13:02:08.100701 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:08.101269 25244 external_mini_cluster.cc:1428] Setting key 394ddbac4069997b71c098994cb19870
I20260808 13:02:08.105616 24309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43: Bootstrap starting.
I20260808 13:02:08.109112 24309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:08.110455 24309 log.cc:824] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:08.113724 24309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43: No bootstrap required, opened a new log
I20260808 13:02:08.114974 24313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:08.103135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45291 (local address 127.0.0.1:44295)
0808 13:02:08.103998 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:08.104008 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:08.104048 (+ 40us) server_negotiation.cc:407] Connection header received
0808 13:02:08.105665 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:08.105686 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:08.106091 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:08.106462 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:08.107027 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:08.108345 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:08.110228 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:08.111258 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:08.111475 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:08.112748 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:08.112755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:08.112769 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:08.113284 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:08.113357 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:08.113380 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:08.113504 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:08.113820 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 13:02:08.113984 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":421,"threads_started":1}
I20260808 13:02:08.118842 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:08.122076 24309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e89e722a1946dfb01f93663eacda43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44295 } }
I20260808 13:02:08.122536 24309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:08.122635 24309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e89e722a1946dfb01f93663eacda43, State: Initialized, Role: FOLLOWER
I20260808 13:02:08.123373 24309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e89e722a1946dfb01f93663eacda43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44295 } }
I20260808 13:02:08.123689 24309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:08.123835 24309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:08.124047 24309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:08.127170 24309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e89e722a1946dfb01f93663eacda43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44295 } }
I20260808 13:02:08.127799 24309 leader_election.cc:302] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56e89e722a1946dfb01f93663eacda43; no voters:
I20260808 13:02:08.128489 24309 leader_election.cc:288] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:08.128685 24322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:08.130213 24322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [term 1 LEADER]: Becoming Leader. State: Replica: 56e89e722a1946dfb01f93663eacda43, State: Running, Role: LEADER
I20260808 13:02:08.130947 24322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e89e722a1946dfb01f93663eacda43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44295 } }
I20260808 13:02:08.131191 24313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:08.123365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37739 (local address 127.0.0.1:44295)
0808 13:02:08.123552 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:08.123559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:08.123939 (+ 380us) server_negotiation.cc:407] Connection header received
0808 13:02:08.124180 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:08.124186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:08.124274 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:08.124389 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:08.124916 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:08.126090 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:08.127569 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:08.128314 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:08.128487 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:08.129919 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:08.129928 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:08.129933 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:08.129965 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:08.130053 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:08.130082 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:08.130221 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:08.130702 (+ 481us) server_negotiation.cc:299] Negotiation successful
0808 13:02:08.130767 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 13:02:08.131253 24309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:08.133874 24325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56e89e722a1946dfb01f93663eacda43. Latest consensus state: current_term: 1 leader_uuid: "56e89e722a1946dfb01f93663eacda43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e89e722a1946dfb01f93663eacda43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44295 } } }
I20260808 13:02:08.134276 24323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56e89e722a1946dfb01f93663eacda43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e89e722a1946dfb01f93663eacda43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44295 } } }
I20260808 13:02:08.134579 24325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:08.134616 24323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:08.136253 24328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:08.141453 24328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:08.153035 24328 catalog_manager.cc:1347] Generated new cluster ID: bc7d54e3d63d49d0bcaf990c0f91d003
I20260808 13:02:08.153169 24328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:08.165479 24328 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:08.166949 24328 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:08.175364 24328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56e89e722a1946dfb01f93663eacda43: Generated new TSK 0
I20260808 13:02:08.176482 24328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:08.189957 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 24220
W20260808 13:02:08.191191 24303 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45291
W20260808 13:02:08.192049 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 24220: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 13:02:08.192246 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 24220
W20260808 13:02:08.218580 24200 connection.cc:535] client connection to 127.0.0.1:44295 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44295): Connection reset by peer (error 104)
2026-08-08T13:02:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2235 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:08 dist-test-slave-tk9z krb5kdc[24346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:08 dist-test-slave-tk9z krb5kdc[24346](info): set up 2 sockets
Aug 08 13:02:08 dist-test-slave-tk9z krb5kdc[24346](info): commencing operation
krb5kdc: starting...
W20260808 13:02:09.533824 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.295s user 0.003s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:09 dist-test-slave-tk9z krb5kdc[24346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 13:02:09.860049 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40785
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:10.217463 24368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:10.217777 24368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:10.217959 24368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:10.225557 24368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:10.225682 24368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:10.225775 24368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:10.225884 24368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:10.225977 24368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:10.236334 24368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40785
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:10.237727 24368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:10.239063 24368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:10.251681 24376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:10.251722 24379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:10.251747 24381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:10.252360 24368 server_base.cc:1034] running on GCE node
I20260808 13:02:10.253234 24368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:10.254742 24368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:10.255981 24368 hybrid_clock.cc:648] HybridClock initialized: now 1786194130255933 us; error 42 us; skew 500 ppm
I20260808 13:02:10.256343 24368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:10 dist-test-slave-tk9z krb5kdc[24346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:10.259698 24368 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 13:02:10.261336 24368 webserver.cc:460] Webserver started at http://127.0.0.1:34759/ using document root <none> and password file <none>
I20260808 13:02:10.262061 24368 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:10.262217 24368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:10.262606 24368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:10.265754 24368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "c892602d0baf43c7a5b89073fde3d02a"
format_stamp: "Formatted at 2026-08-08 13:02:10 on dist-test-slave-tk9z"
server_key: "eba6a8adf36a5d18255a9b84d2b300d4"
server_key_iv: "10c4ab0b6e7b156d7b1d48f05da81156"
server_key_version: "encryptionkey@0"
I20260808 13:02:10.266667 24368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "c892602d0baf43c7a5b89073fde3d02a"
format_stamp: "Formatted at 2026-08-08 13:02:10 on dist-test-slave-tk9z"
server_key: "eba6a8adf36a5d18255a9b84d2b300d4"
server_key_iv: "10c4ab0b6e7b156d7b1d48f05da81156"
server_key_version: "encryptionkey@0"
I20260808 13:02:10.271973 24368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 13:02:10.275508 24393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:10.276922 24368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 13:02:10.277127 24368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "c892602d0baf43c7a5b89073fde3d02a"
format_stamp: "Formatted at 2026-08-08 13:02:10 on dist-test-slave-tk9z"
server_key: "eba6a8adf36a5d18255a9b84d2b300d4"
server_key_iv: "10c4ab0b6e7b156d7b1d48f05da81156"
server_key_version: "encryptionkey@0"
I20260808 13:02:10.277343 24368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:10.283346 24368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:10.287384 24368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:10.287704 24368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:10.307078 24368 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40785
I20260808 13:02:10.307127 24454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40785 every 8 connection(s)
I20260808 13:02:10.308593 24368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:10.312484 24455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:10.312537 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 24368
I20260808 13:02:10.312731 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:10.313220 25244 external_mini_cluster.cc:1428] Setting key c18c8287d94077320f70b1aef8992afe
I20260808 13:02:10.321204 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:10.315101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34593 (local address 127.0.0.1:40785)
0808 13:02:10.315954 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:10.315969 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:10.316016 (+ 47us) server_negotiation.cc:407] Connection header received
0808 13:02:10.317681 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:10.317704 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:10.318052 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:10.318496 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:10.319581 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:10.319587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:10.319600 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:10.319613 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:10.319674 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:10.319695 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:10.319958 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:10.320106 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 13:02:10.320306 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":383,"threads_started":1}
I20260808 13:02:10.325055 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:10.326361 24455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a: Bootstrap starting.
I20260808 13:02:10.329849 24455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:10.331097 24455 log.cc:824] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:10.332104 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:10.329487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52229 (local address 127.0.0.1:40785)
0808 13:02:10.329667 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:10.329674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:10.330076 (+ 402us) server_negotiation.cc:407] Connection header received
0808 13:02:10.330291 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:10.330295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:10.330344 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:10.330482 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:10.331447 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:10.331452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:10.331456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:10.331476 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:10.331521 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:10.331536 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:10.331666 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:10.331766 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 13:02:10.331829 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 13:02:10.334133 24455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a: No bootstrap required, opened a new log
I20260808 13:02:10.342305 24455 raft_consensus.cc:348] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c892602d0baf43c7a5b89073fde3d02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40785 } }
I20260808 13:02:10.342787 24455 raft_consensus.cc:374] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:10.342911 24455 raft_consensus.cc:729] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c892602d0baf43c7a5b89073fde3d02a, State: Initialized, Role: FOLLOWER
I20260808 13:02:10.343555 24455 consensus_queue.cc:260] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c892602d0baf43c7a5b89073fde3d02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40785 } }
I20260808 13:02:10.343802 24455 raft_consensus.cc:388] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:10.343964 24455 raft_consensus.cc:482] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:10.344147 24455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:10.346488 24455 raft_consensus.cc:504] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c892602d0baf43c7a5b89073fde3d02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40785 } }
I20260808 13:02:10.347034 24455 leader_election.cc:302] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c892602d0baf43c7a5b89073fde3d02a; no voters:
I20260808 13:02:10.347745 24455 leader_election.cc:288] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:10.347903 24467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:10.348822 24467 raft_consensus.cc:686] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [term 1 LEADER]: Becoming Leader. State: Replica: c892602d0baf43c7a5b89073fde3d02a, State: Running, Role: LEADER
I20260808 13:02:10.349355 24467 consensus_queue.cc:237] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c892602d0baf43c7a5b89073fde3d02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40785 } }
I20260808 13:02:10.349987 24455 sys_catalog.cc:564] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:10.352350 24468 sys_catalog.cc:455] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c892602d0baf43c7a5b89073fde3d02a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c892602d0baf43c7a5b89073fde3d02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40785 } } }
I20260808 13:02:10.352447 24470 sys_catalog.cc:455] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c892602d0baf43c7a5b89073fde3d02a. Latest consensus state: current_term: 1 leader_uuid: "c892602d0baf43c7a5b89073fde3d02a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c892602d0baf43c7a5b89073fde3d02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40785 } } }
I20260808 13:02:10.352906 24468 sys_catalog.cc:458] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:10.352914 24470 sys_catalog.cc:458] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:10.354008 24482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:10.360288 24482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:10.369680 24482 catalog_manager.cc:1347] Generated new cluster ID: c9c1e9f03fa643869a17e0d9d0861ac1
I20260808 13:02:10.369817 24482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:10.408713 24482 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:10.409195 24482 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:10.418738 24482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c892602d0baf43c7a5b89073fde3d02a: Generated new TSK 0
I20260808 13:02:10.419643 24482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:10.432005 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 24368
W20260808 13:02:10.433171 24449 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34593
W20260808 13:02:10.433888 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 24368: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 13:02:10.434052 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 24368
2026-08-08T13:02:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2240 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:10 dist-test-slave-tk9z krb5kdc[24496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:10 dist-test-slave-tk9z krb5kdc[24496](info): set up 2 sockets
Aug 08 13:02:10 dist-test-slave-tk9z krb5kdc[24496](info): commencing operation
krb5kdc: starting...
W20260808 13:02:11.785554 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.307s user 0.003s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:11 dist-test-slave-tk9z krb5kdc[24496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 13:02:12.120401 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:12.481338 24515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:12.481657 24515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:12.481837 24515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:12.489348 24515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:12.489512 24515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:12.489619 24515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:12.489737 24515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:12.489820 24515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:12.500191 24515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:12.501590 24515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:12.503054 24515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:12.516387 24524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:12.516337 24525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:12.516280 24527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:12.517307 24515 server_base.cc:1034] running on GCE node
I20260808 13:02:12.518244 24515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:12.519765 24515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:12.521016 24515 hybrid_clock.cc:648] HybridClock initialized: now 1786194132520967 us; error 46 us; skew 500 ppm
I20260808 13:02:12.521391 24515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:12 dist-test-slave-tk9z krb5kdc[24496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:12.524917 24515 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 13:02:12.526716 24515 webserver.cc:460] Webserver started at http://127.0.0.1:43175/ using document root <none> and password file <none>
I20260808 13:02:12.527525 24515 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:12.527702 24515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:12.528051 24515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:12.531546 24515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "6c112169f7df4fb6b31028ccb9da29f6"
format_stamp: "Formatted at 2026-08-08 13:02:12 on dist-test-slave-tk9z"
server_key: "b6eb76026248aaa96832f7ff63238902"
server_key_iv: "7e45625a7f2e8acf3dda3b9779fb37cc"
server_key_version: "encryptionkey@0"
I20260808 13:02:12.532449 24515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "6c112169f7df4fb6b31028ccb9da29f6"
format_stamp: "Formatted at 2026-08-08 13:02:12 on dist-test-slave-tk9z"
server_key: "b6eb76026248aaa96832f7ff63238902"
server_key_iv: "7e45625a7f2e8acf3dda3b9779fb37cc"
server_key_version: "encryptionkey@0"
I20260808 13:02:12.538241 24515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 13:02:12.542124 24536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:12.543686 24515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:02:12.543941 24515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "6c112169f7df4fb6b31028ccb9da29f6"
format_stamp: "Formatted at 2026-08-08 13:02:12 on dist-test-slave-tk9z"
server_key: "b6eb76026248aaa96832f7ff63238902"
server_key_iv: "7e45625a7f2e8acf3dda3b9779fb37cc"
server_key_version: "encryptionkey@0"
I20260808 13:02:12.544190 24515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:12.588881 24515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:12.594403 24515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:12.594775 24515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:12.616250 24515 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46539
I20260808 13:02:12.616282 24611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46539 every 8 connection(s)
I20260808 13:02:12.617883 24515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:12.622340 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:12.624924 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 24515
I20260808 13:02:12.625146 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:12.625783 25244 external_mini_cluster.cc:1428] Setting key 9cc15c28486280834218ddd54909a328
I20260808 13:02:12.637316 24612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6: Bootstrap starting.
I20260808 13:02:12.642084 24612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6: Neither blocks nor log segments found. Creating new log.
Aug 08 13:02:12 dist-test-slave-tk9z krb5kdc[24496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 13:02:12.643548 24612 log.cc:824] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:12.647661 24612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6: No bootstrap required, opened a new log
I20260808 13:02:12.654241 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:12.628040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59683 (local address 127.0.0.1:46539)
0808 13:02:12.629046 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:12.629064 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:12.629118 (+ 54us) server_negotiation.cc:407] Connection header received
0808 13:02:12.631197 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:12.631224 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:12.631698 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:12.632072 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:12.634194 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:12.635884 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:12.637954 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:12.638832 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:12.639000 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:12.644554 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:12.644563 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:12.644579 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:12.644651 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:12.649068 (+ 4417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:12.649955 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:12.649971 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:12.649981 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:12.650208 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:12.650682 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:12.650688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:12.650692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:12.651529 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:12.651865 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:12.652212 (+ 347us) server_negotiation.cc:299] Negotiation successful
0808 13:02:12.652724 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":468,"threads_started":1}
I20260808 13:02:12.658286 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:12.658771 24612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c112169f7df4fb6b31028ccb9da29f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46539 } }
I20260808 13:02:12.659266 24612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:12.659394 24612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c112169f7df4fb6b31028ccb9da29f6, State: Initialized, Role: FOLLOWER
I20260808 13:02:12.660076 24612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c112169f7df4fb6b31028ccb9da29f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46539 } }
I20260808 13:02:12.660392 24612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:12.660544 24612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:12.660732 24612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:12.664610 24612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c112169f7df4fb6b31028ccb9da29f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46539 } }
I20260808 13:02:12.665272 24612 leader_election.cc:302] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c112169f7df4fb6b31028ccb9da29f6; no voters:
I20260808 13:02:12.666035 24612 leader_election.cc:288] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:12.666147 24623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:12.669241 24612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:12.671022 24623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [term 1 LEADER]: Becoming Leader. State: Replica: 6c112169f7df4fb6b31028ccb9da29f6, State: Running, Role: LEADER
I20260808 13:02:12.671866 24623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c112169f7df4fb6b31028ccb9da29f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46539 } }
I20260808 13:02:12.675921 24625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c112169f7df4fb6b31028ccb9da29f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c112169f7df4fb6b31028ccb9da29f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46539 } } }
I20260808 13:02:12.676350 24625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:12.677421 24634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:12.678864 24626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c112169f7df4fb6b31028ccb9da29f6. Latest consensus state: current_term: 1 leader_uuid: "6c112169f7df4fb6b31028ccb9da29f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c112169f7df4fb6b31028ccb9da29f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46539 } } }
I20260808 13:02:12.679983 24626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:12.681504 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:12.664235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52411 (local address 127.0.0.1:46539)
0808 13:02:12.664473 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:12.664480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:12.664890 (+ 410us) server_negotiation.cc:407] Connection header received
0808 13:02:12.665103 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:12.665109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:12.665193 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:12.665310 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:12.667189 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:12.667999 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:12.669335 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:12.669850 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:12.670004 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:12.672846 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:12.672851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:12.672854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:12.672885 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:12.677283 (+ 4398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:12.678255 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:12.678259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:12.678263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:12.678512 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:12.679100 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:12.679106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:12.679112 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:12.679551 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:12.680965 (+ 1414us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:12.681090 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 13:02:12.681265 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 13:02:12.689949 24634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:12.699952 24634 catalog_manager.cc:1347] Generated new cluster ID: 02386fc166874e8d9958142be834f2c6
I20260808 13:02:12.700145 24634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:12.715838 24634 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:12.717960 24634 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:12.727651 24634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c112169f7df4fb6b31028ccb9da29f6: Generated new TSK 0
I20260808 13:02:12.728550 24634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:12.740139 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 24515
I20260808 13:02:12.741379 24606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:12.835480 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 24515
2026-08-08T13:02:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:12 dist-test-slave-tk9z krb5kdc[24651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:12 dist-test-slave-tk9z krb5kdc[24651](info): set up 2 sockets
Aug 08 13:02:12 dist-test-slave-tk9z krb5kdc[24651](info): commencing operation
krb5kdc: starting...
W20260808 13:02:14.218544 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.337s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:14 dist-test-slave-tk9z krb5kdc[24651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 13:02:14.549932 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:14.918874 24671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:14.919243 24671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:14.919438 24671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:14.927050 24671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:14.927213 24671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:14.927326 24671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:14.927441 24671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:14.927529 24671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:14.937769 24671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:14.939236 24671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:14.940609 24671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:14.952940 24680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:14.953003 24684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:14.953258 24678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:14.954196 24671 server_base.cc:1034] running on GCE node
I20260808 13:02:14.955066 24671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:14.956568 24671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:14.957824 24671 hybrid_clock.cc:648] HybridClock initialized: now 1786194134957831 us; error 100 us; skew 500 ppm
I20260808 13:02:14.958194 24671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:14 dist-test-slave-tk9z krb5kdc[24651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:14.961529 24671 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 13:02:14.963280 24671 webserver.cc:460] Webserver started at http://127.0.0.1:39035/ using document root <none> and password file <none>
I20260808 13:02:14.964021 24671 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:14.964175 24671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:14.964506 24671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:14.967747 24671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "acfdcb7e8c574ef796c91229c8b73503"
format_stamp: "Formatted at 2026-08-08 13:02:14 on dist-test-slave-tk9z"
server_key: "8eb1c9f86489791bbe77c8e05b5694c5"
server_key_iv: "a835b07d9e25258889815a63c0bc05c2"
server_key_version: "encryptionkey@0"
I20260808 13:02:14.968573 24671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "acfdcb7e8c574ef796c91229c8b73503"
format_stamp: "Formatted at 2026-08-08 13:02:14 on dist-test-slave-tk9z"
server_key: "8eb1c9f86489791bbe77c8e05b5694c5"
server_key_iv: "a835b07d9e25258889815a63c0bc05c2"
server_key_version: "encryptionkey@0"
I20260808 13:02:14.973896 24671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 13:02:14.977484 24693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:14.978840 24671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:02:14.979029 24671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "acfdcb7e8c574ef796c91229c8b73503"
format_stamp: "Formatted at 2026-08-08 13:02:14 on dist-test-slave-tk9z"
server_key: "8eb1c9f86489791bbe77c8e05b5694c5"
server_key_iv: "a835b07d9e25258889815a63c0bc05c2"
server_key_version: "encryptionkey@0"
I20260808 13:02:14.979238 24671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:15.015311 24671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:15.020046 24671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:15.020404 24671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:15.040150 24671 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43527
I20260808 13:02:15.040190 24761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43527 every 8 connection(s)
I20260808 13:02:15.041666 24671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:15.043380 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 24671
I20260808 13:02:15.043619 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:15.044169 25244 external_mini_cluster.cc:1428] Setting key a49be3d24ea35331945de2ca717cbeef
I20260808 13:02:15.046561 24762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:15.057971 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:15.046072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46217 (local address 127.0.0.1:43527)
0808 13:02:15.046931 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:15.046942 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:15.046983 (+ 41us) server_negotiation.cc:407] Connection header received
0808 13:02:15.048645 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:15.048667 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:15.049058 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:15.049401 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:15.050036 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:15.051516 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:15.053398 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:15.054061 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:15.054191 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:15.055593 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:15.055600 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:15.055617 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:15.056171 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:15.056251 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:15.056280 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:15.056384 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:15.056637 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 13:02:15.056943 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":412,"threads_started":1}
I20260808 13:02:15.059512 24762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503: Bootstrap starting.
I20260808 13:02:15.062036 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:15.062991 24762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:15.064003 24762 log.cc:824] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:15.067587 24762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503: No bootstrap required, opened a new log
I20260808 13:02:15.073323 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:15.067042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35487 (local address 127.0.0.1:43527)
0808 13:02:15.067238 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:15.067245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:15.067667 (+ 422us) server_negotiation.cc:407] Connection header received
0808 13:02:15.067910 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:15.067914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:15.067974 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:15.068101 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:15.068634 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:15.069399 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:15.070786 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:15.071258 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:15.071404 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:15.072520 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:15.072525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:15.072531 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:15.072548 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:15.072594 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:15.072613 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:15.072722 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:15.072899 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 13:02:15.072997 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 13:02:15.075971 24762 raft_consensus.cc:348] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfdcb7e8c574ef796c91229c8b73503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43527 } }
I20260808 13:02:15.076382 24762 raft_consensus.cc:374] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:15.076494 24762 raft_consensus.cc:729] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acfdcb7e8c574ef796c91229c8b73503, State: Initialized, Role: FOLLOWER
I20260808 13:02:15.077066 24762 consensus_queue.cc:260] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfdcb7e8c574ef796c91229c8b73503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43527 } }
I20260808 13:02:15.077325 24762 raft_consensus.cc:388] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:15.077481 24762 raft_consensus.cc:482] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:15.077672 24762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:15.080044 24762 raft_consensus.cc:504] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfdcb7e8c574ef796c91229c8b73503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43527 } }
I20260808 13:02:15.080662 24762 leader_election.cc:302] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acfdcb7e8c574ef796c91229c8b73503; no voters:
I20260808 13:02:15.081331 24762 leader_election.cc:288] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:15.081453 24775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:15.082175 24775 raft_consensus.cc:686] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [term 1 LEADER]: Becoming Leader. State: Replica: acfdcb7e8c574ef796c91229c8b73503, State: Running, Role: LEADER
I20260808 13:02:15.082841 24775 consensus_queue.cc:237] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfdcb7e8c574ef796c91229c8b73503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43527 } }
I20260808 13:02:15.083662 24762 sys_catalog.cc:564] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:15.086285 24777 sys_catalog.cc:455] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acfdcb7e8c574ef796c91229c8b73503. Latest consensus state: current_term: 1 leader_uuid: "acfdcb7e8c574ef796c91229c8b73503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfdcb7e8c574ef796c91229c8b73503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43527 } } }
I20260808 13:02:15.086452 24776 sys_catalog.cc:455] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acfdcb7e8c574ef796c91229c8b73503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfdcb7e8c574ef796c91229c8b73503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43527 } } }
I20260808 13:02:15.086763 24777 sys_catalog.cc:458] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:15.086763 24776 sys_catalog.cc:458] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:15.087723 24783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:15.093031 24783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:15.102072 24783 catalog_manager.cc:1347] Generated new cluster ID: ef4a1c9d9a434e329ab0c1febc57a993
I20260808 13:02:15.102205 24783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:15.118206 24783 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:15.120272 24783 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:15.130470 24783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acfdcb7e8c574ef796c91229c8b73503: Generated new TSK 0
I20260808 13:02:15.131618 24783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:15.143638 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 24671
W20260808 13:02:15.144810 24756 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46217
W20260808 13:02:15.145664 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 24671: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 13:02:15.145830 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 24671
2026-08-08T13:02:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:15 dist-test-slave-tk9z krb5kdc[24804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:15 dist-test-slave-tk9z krb5kdc[24804](info): set up 2 sockets
Aug 08 13:02:15 dist-test-slave-tk9z krb5kdc[24804](info): commencing operation
krb5kdc: starting...
W20260808 13:02:16.527070 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.333s user 0.014s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:16 dist-test-slave-tk9z krb5kdc[24804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 13:02:16.873682 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:17.237279 24824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:17.237594 24824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:17.237761 24824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:17.245246 24824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:17.245388 24824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:17.245486 24824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:17.245581 24824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:17.245651 24824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:17.256507 24824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:17.257936 24824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:17.259377 24824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:17.272331 24833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:17.272667 24835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:17.272701 24832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:17.273579 24824 server_base.cc:1034] running on GCE node
I20260808 13:02:17.274570 24824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:17.276173 24824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:17.277436 24824 hybrid_clock.cc:648] HybridClock initialized: now 1786194137277386 us; error 42 us; skew 500 ppm
I20260808 13:02:17.277850 24824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:17 dist-test-slave-tk9z krb5kdc[24804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:17.286208 24824 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 13:02:17.288115 24824 webserver.cc:460] Webserver started at http://127.0.0.1:45407/ using document root <none> and password file <none>
I20260808 13:02:17.288858 24824 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:17.289016 24824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:17.289345 24824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:17.292629 24824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "e39162fbdc654466bb74665fa2d1fbcc"
format_stamp: "Formatted at 2026-08-08 13:02:17 on dist-test-slave-tk9z"
server_key: "0ecae738751f80c125748676855b8b65"
server_key_iv: "e7ff7c3d44b0df9c6a775800d6b94c83"
server_key_version: "encryptionkey@0"
I20260808 13:02:17.293467 24824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "e39162fbdc654466bb74665fa2d1fbcc"
format_stamp: "Formatted at 2026-08-08 13:02:17 on dist-test-slave-tk9z"
server_key: "0ecae738751f80c125748676855b8b65"
server_key_iv: "e7ff7c3d44b0df9c6a775800d6b94c83"
server_key_version: "encryptionkey@0"
I20260808 13:02:17.298902 24824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 13:02:17.302529 24843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:17.303889 24824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 13:02:17.304105 24824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "e39162fbdc654466bb74665fa2d1fbcc"
format_stamp: "Formatted at 2026-08-08 13:02:17 on dist-test-slave-tk9z"
server_key: "0ecae738751f80c125748676855b8b65"
server_key_iv: "e7ff7c3d44b0df9c6a775800d6b94c83"
server_key_version: "encryptionkey@0"
I20260808 13:02:17.304328 24824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:17.310515 24824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:17.314790 24824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:17.315102 24824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:17.334964 24824 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41181
I20260808 13:02:17.335057 24902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41181 every 8 connection(s)
I20260808 13:02:17.336409 24824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:17.340296 24903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:17.346546 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 24824
I20260808 13:02:17.346740 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:17.347285 25244 external_mini_cluster.cc:1428] Setting key 24e0cd125f35aaeb0f5eac5caf71a14f
I20260808 13:02:17.351861 24903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc: Bootstrap starting.
I20260808 13:02:17.355563 24909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:17.349201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43909 (local address 127.0.0.1:41181)
0808 13:02:17.350167 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:17.350179 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:17.350221 (+ 42us) server_negotiation.cc:407] Connection header received
0808 13:02:17.352081 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:17.352104 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:17.352428 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:17.352743 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:17.353928 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:17.353934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:17.353958 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:17.353973 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:17.354037 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:17.354062 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:17.354236 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:17.354450 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 13:02:17.354668 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":473,"threads_started":1}
I20260808 13:02:17.355540 24903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:17.356837 24903 log.cc:824] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:17.359637 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:17.360107 24903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc: No bootstrap required, opened a new log
I20260808 13:02:17.366694 24909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:17.364253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34479 (local address 127.0.0.1:41181)
0808 13:02:17.364473 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:17.364479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:17.364743 (+ 264us) server_negotiation.cc:407] Connection header received
0808 13:02:17.364892 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:17.364898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:17.364965 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:17.365085 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:17.365996 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:17.366002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:17.366007 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:17.366025 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:17.366080 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:17.366099 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:17.366240 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:17.366291 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 13:02:17.366351 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 13:02:17.368814 24903 raft_consensus.cc:348] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39162fbdc654466bb74665fa2d1fbcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } }
I20260808 13:02:17.369181 24903 raft_consensus.cc:374] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:17.369263 24903 raft_consensus.cc:729] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39162fbdc654466bb74665fa2d1fbcc, State: Initialized, Role: FOLLOWER
I20260808 13:02:17.369839 24903 consensus_queue.cc:260] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39162fbdc654466bb74665fa2d1fbcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } }
I20260808 13:02:17.370075 24903 raft_consensus.cc:388] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:17.370178 24903 raft_consensus.cc:482] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:17.370330 24903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:17.372586 24903 raft_consensus.cc:504] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39162fbdc654466bb74665fa2d1fbcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } }
I20260808 13:02:17.373049 24903 leader_election.cc:302] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e39162fbdc654466bb74665fa2d1fbcc; no voters:
I20260808 13:02:17.373636 24903 leader_election.cc:288] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:17.373848 24919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:17.375008 24919 raft_consensus.cc:686] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [term 1 LEADER]: Becoming Leader. State: Replica: e39162fbdc654466bb74665fa2d1fbcc, State: Running, Role: LEADER
I20260808 13:02:17.375573 24919 consensus_queue.cc:237] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39162fbdc654466bb74665fa2d1fbcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } }
I20260808 13:02:17.376024 24903 sys_catalog.cc:564] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:17.378296 24922 sys_catalog.cc:455] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e39162fbdc654466bb74665fa2d1fbcc. Latest consensus state: current_term: 1 leader_uuid: "e39162fbdc654466bb74665fa2d1fbcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39162fbdc654466bb74665fa2d1fbcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } } }
I20260808 13:02:17.378799 24922 sys_catalog.cc:458] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:17.378670 24920 sys_catalog.cc:455] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e39162fbdc654466bb74665fa2d1fbcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39162fbdc654466bb74665fa2d1fbcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } } }
I20260808 13:02:17.379072 24920 sys_catalog.cc:458] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:17.379944 24928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:17.385879 24928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:17.395539 24928 catalog_manager.cc:1347] Generated new cluster ID: f4394282cde144a9822c4b9a713f90d5
I20260808 13:02:17.395730 24928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:17.412776 24928 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:17.413298 24928 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:17.419337 24928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e39162fbdc654466bb74665fa2d1fbcc: Generated new TSK 0
I20260808 13:02:17.420333 24928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:17.434655 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 24824
W20260808 13:02:17.435827 24898 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43909
W20260808 13:02:17.436546 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 24824: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 13:02:17.436682 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 24824
2026-08-08T13:02:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:17 dist-test-slave-tk9z krb5kdc[24949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:17 dist-test-slave-tk9z krb5kdc[24949](info): set up 2 sockets
Aug 08 13:02:17 dist-test-slave-tk9z krb5kdc[24949](info): commencing operation
krb5kdc: starting...
W20260808 13:02:18.791888 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.304s user 0.001s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:19 dist-test-slave-tk9z krb5kdc[24949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 13:02:19.126616 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:19.483050 24968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:19.483364 24968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:19.483536 24968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:19.490764 24968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:19.490900 24968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:19.491009 24968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:19.491117 24968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:19.491199 24968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:19.500914 24968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:19.502216 24968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:19.503865 24968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:19.516863 24973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:19.517048 24978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:19.516903 24975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:19.518482 24968 server_base.cc:1034] running on GCE node
I20260808 13:02:19.519335 24968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:19.520864 24968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:19.522094 24968 hybrid_clock.cc:648] HybridClock initialized: now 1786194139522034 us; error 57 us; skew 500 ppm
I20260808 13:02:19.522538 24968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:19 dist-test-slave-tk9z krb5kdc[24949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:19.525833 24968 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 13:02:19.527415 24968 webserver.cc:460] Webserver started at http://127.0.0.1:36719/ using document root <none> and password file <none>
I20260808 13:02:19.528142 24968 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:19.528250 24968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:19.528517 24968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:19.531572 24968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "29b88cb057474c37b35ceeda13a60049"
format_stamp: "Formatted at 2026-08-08 13:02:19 on dist-test-slave-tk9z"
server_key: "d5192b12a3d63cf021ccb93eeb69f110"
server_key_iv: "0f0fa29f1d8d76aa6edadd45a48b2055"
server_key_version: "encryptionkey@0"
I20260808 13:02:19.532342 24968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "29b88cb057474c37b35ceeda13a60049"
format_stamp: "Formatted at 2026-08-08 13:02:19 on dist-test-slave-tk9z"
server_key: "d5192b12a3d63cf021ccb93eeb69f110"
server_key_iv: "0f0fa29f1d8d76aa6edadd45a48b2055"
server_key_version: "encryptionkey@0"
I20260808 13:02:19.537561 24968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 13:02:19.541157 24988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:19.542728 24968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 13:02:19.542924 24968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "29b88cb057474c37b35ceeda13a60049"
format_stamp: "Formatted at 2026-08-08 13:02:19 on dist-test-slave-tk9z"
server_key: "d5192b12a3d63cf021ccb93eeb69f110"
server_key_iv: "0f0fa29f1d8d76aa6edadd45a48b2055"
server_key_version: "encryptionkey@0"
I20260808 13:02:19.543121 24968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:19.557245 24968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:19.561595 24968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:19.561906 24968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:19.582350 24968 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38679
I20260808 13:02:19.582415 25054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38679 every 8 connection(s)
I20260808 13:02:19.583953 24968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:19.588069 25056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:19.589529 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 24968
I20260808 13:02:19.589730 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:19.590330 25244 external_mini_cluster.cc:1428] Setting key ff33013889fc16da0be69314c143db3a
I20260808 13:02:19.603328 25056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049: Bootstrap starting.
Aug 08 13:02:19 dist-test-slave-tk9z krb5kdc[24949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 13:02:19.607113 25056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:19.608500 25056 log.cc:824] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:19.611825 25056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049: No bootstrap required, opened a new log
I20260808 13:02:19.615685 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:19.592421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:43975 (local address 127.0.0.1:38679)
0808 13:02:19.593251 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:19.593263 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:19.593305 (+ 42us) server_negotiation.cc:407] Connection header received
0808 13:02:19.595067 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:19.595094 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:19.595494 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:19.595823 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:19.597657 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:19.599018 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:19.600909 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:19.601549 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:19.601672 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:19.606746 (+ 5074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:19.606754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:19.606768 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:19.606809 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:19.611033 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:19.611911 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:19.611924 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:19.611955 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:19.612186 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:19.612697 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:19.612702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:19.612705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:19.613244 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:19.613633 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:19.614136 (+ 503us) server_negotiation.cc:299] Negotiation successful
0808 13:02:19.614542 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":383,"threads_started":1}
I20260808 13:02:19.620024 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:19.620715 25056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b88cb057474c37b35ceeda13a60049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } }
I20260808 13:02:19.621117 25056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:19.621230 25056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b88cb057474c37b35ceeda13a60049, State: Initialized, Role: FOLLOWER
I20260808 13:02:19.621820 25056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b88cb057474c37b35ceeda13a60049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } }
I20260808 13:02:19.622097 25056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:19.622249 25056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:19.622471 25056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:19.625401 25056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b88cb057474c37b35ceeda13a60049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } }
I20260808 13:02:19.626102 25056 leader_election.cc:302] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29b88cb057474c37b35ceeda13a60049; no voters:
I20260808 13:02:19.626814 25056 leader_election.cc:288] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:19.626895 25071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:19.629081 25056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:19.629500 25071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [term 1 LEADER]: Becoming Leader. State: Replica: 29b88cb057474c37b35ceeda13a60049, State: Running, Role: LEADER
I20260808 13:02:19.630112 25071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b88cb057474c37b35ceeda13a60049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } }
I20260808 13:02:19.632997 25072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29b88cb057474c37b35ceeda13a60049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b88cb057474c37b35ceeda13a60049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } } }
I20260808 13:02:19.633389 25072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:19.633695 25073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b88cb057474c37b35ceeda13a60049. Latest consensus state: current_term: 1 leader_uuid: "29b88cb057474c37b35ceeda13a60049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b88cb057474c37b35ceeda13a60049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } } }
I20260808 13:02:19.634049 25073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:19.648227 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:19.624871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:41443 (local address 127.0.0.1:38679)
0808 13:02:19.625101 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:19.625109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:19.625665 (+ 556us) server_negotiation.cc:407] Connection header received
0808 13:02:19.625951 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:19.625957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:19.626023 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:19.626104 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:19.628303 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:19.629787 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:19.632218 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:19.633153 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:19.633305 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:19.637514 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:19.637524 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:19.637533 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:19.637592 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:19.642534 (+ 4942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:19.643946 (+ 1412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:19.643955 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:19.643962 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:19.644162 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:19.644925 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:19.644934 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:19.644940 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:19.645534 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:19.645912 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:19.647344 (+ 1432us) server_negotiation.cc:299] Negotiation successful
0808 13:02:19.647593 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 13:02:19.650983 25078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:19.656941 25078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:19.666038 25078 catalog_manager.cc:1347] Generated new cluster ID: eeec165587a34601b487fa059e5d861c
I20260808 13:02:19.666177 25078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:19.679914 25078 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:19.681242 25078 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:19.686957 25078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29b88cb057474c37b35ceeda13a60049: Generated new TSK 0
I20260808 13:02:19.687841 25078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:19.695305 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 24968
I20260808 13:02:19.696559 25049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:19.795006 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 24968
W20260808 13:02:19.823393 24947 connection.cc:535] client connection to 127.0.0.1:38679 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38679): Connection reset by peer (error 104)
2026-08-08T13:02:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:19 dist-test-slave-tk9z krb5kdc[25104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:19 dist-test-slave-tk9z krb5kdc[25104](info): set up 2 sockets
Aug 08 13:02:19 dist-test-slave-tk9z krb5kdc[25104](info): commencing operation
krb5kdc: starting...
W20260808 13:02:21.132578 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.289s user 0.004s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:21 dist-test-slave-tk9z krb5kdc[25104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 13:02:21.475948 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:21.839387 25120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:21.839682 25120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:21.839840 25120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:21.847461 25120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:21.847589 25120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:21.847681 25120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:21.847770 25120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:21.847865 25120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:21.857955 25120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:21.859409 25120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:21.860805 25120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:21.873998 25131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:21.873968 25128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:21.873922 25127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:21.874784 25120 server_base.cc:1034] running on GCE node
I20260808 13:02:21.875728 25120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:21.877403 25120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:21.878690 25120 hybrid_clock.cc:648] HybridClock initialized: now 1786194141878621 us; error 48 us; skew 500 ppm
I20260808 13:02:21.879148 25120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:21 dist-test-slave-tk9z krb5kdc[25104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:21.882736 25120 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 13:02:21.884461 25120 webserver.cc:460] Webserver started at http://127.0.0.1:32817/ using document root <none> and password file <none>
I20260808 13:02:21.885228 25120 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:21.885392 25120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:21.885807 25120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:21.889299 25120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "6aa9175577904d98910dc22b0f1c6348"
format_stamp: "Formatted at 2026-08-08 13:02:21 on dist-test-slave-tk9z"
server_key: "77f4496ff35cdaecbbfd4ce1077a1ff4"
server_key_iv: "8537df294d9e47a84558c94b3da34766"
server_key_version: "encryptionkey@0"
I20260808 13:02:21.890182 25120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "6aa9175577904d98910dc22b0f1c6348"
format_stamp: "Formatted at 2026-08-08 13:02:21 on dist-test-slave-tk9z"
server_key: "77f4496ff35cdaecbbfd4ce1077a1ff4"
server_key_iv: "8537df294d9e47a84558c94b3da34766"
server_key_version: "encryptionkey@0"
I20260808 13:02:21.896786 25120 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260808 13:02:21.900949 25145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:21.902454 25120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 13:02:21.902694 25120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "6aa9175577904d98910dc22b0f1c6348"
format_stamp: "Formatted at 2026-08-08 13:02:21 on dist-test-slave-tk9z"
server_key: "77f4496ff35cdaecbbfd4ce1077a1ff4"
server_key_iv: "8537df294d9e47a84558c94b3da34766"
server_key_version: "encryptionkey@0"
I20260808 13:02:21.902932 25120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:21.949090 25120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:21.954102 25120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:21.954468 25120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:21.975086 25120 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39345
I20260808 13:02:21.975117 25213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39345 every 8 connection(s)
I20260808 13:02:21.976749 25120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:21.981104 25214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:21.981253 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 25120
I20260808 13:02:21.981455 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:21.982102 25244 external_mini_cluster.cc:1428] Setting key 5dde6345d976f0c691d766cb2d5035de
I20260808 13:02:21.995954 25214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348: Bootstrap starting.
I20260808 13:02:21.997870 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:21.984233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:38143 (local address 127.0.0.1:39345)
0808 13:02:21.985184 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:21.985197 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:21.985241 (+ 44us) server_negotiation.cc:407] Connection header received
0808 13:02:21.987869 (+ 2628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:21.987896 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:21.988366 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:21.988726 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:21.989353 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:21.990728 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:21.992792 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:21.993592 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:21.993751 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:21.995285 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:21.995293 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:21.995309 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:21.995953 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:21.996084 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:21.996120 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:21.996264 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:21.996597 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 13:02:21.996825 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":450,"threads_started":1}
I20260808 13:02:21.999635 25214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:22.000990 25214 log.cc:824] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:22.002463 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:22.004163 25214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348: No bootstrap required, opened a new log
I20260808 13:02:22.013917 25214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa9175577904d98910dc22b0f1c6348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39345 } }
I20260808 13:02:22.014564 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:22.007300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:41411 (local address 127.0.0.1:39345)
0808 13:02:22.007532 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:22.007542 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:22.008118 (+ 576us) server_negotiation.cc:407] Connection header received
0808 13:02:22.008329 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:22.008336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:22.008419 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:22.008534 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:22.009058 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:22.010243 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:22.011660 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:22.012402 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:22.012539 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:22.013682 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:22.013689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:22.013693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:22.013715 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:22.013798 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:22.013819 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:22.013964 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:22.014137 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 13:02:22.014200 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 13:02:22.014588 25214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:22.014726 25214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aa9175577904d98910dc22b0f1c6348, State: Initialized, Role: FOLLOWER
I20260808 13:02:22.015455 25214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa9175577904d98910dc22b0f1c6348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39345 } }
I20260808 13:02:22.015748 25214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:22.015935 25214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:22.016145 25214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:22.018748 25214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa9175577904d98910dc22b0f1c6348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39345 } }
I20260808 13:02:22.019260 25214 leader_election.cc:302] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aa9175577904d98910dc22b0f1c6348; no voters:
I20260808 13:02:22.019884 25214 leader_election.cc:288] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:22.020102 25230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:22.020959 25230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [term 1 LEADER]: Becoming Leader. State: Replica: 6aa9175577904d98910dc22b0f1c6348, State: Running, Role: LEADER
I20260808 13:02:22.021651 25230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa9175577904d98910dc22b0f1c6348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39345 } }
I20260808 13:02:22.022526 25214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:22.025259 25231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aa9175577904d98910dc22b0f1c6348" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa9175577904d98910dc22b0f1c6348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39345 } } }
I20260808 13:02:22.025336 25232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aa9175577904d98910dc22b0f1c6348. Latest consensus state: current_term: 1 leader_uuid: "6aa9175577904d98910dc22b0f1c6348" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa9175577904d98910dc22b0f1c6348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39345 } } }
I20260808 13:02:22.025761 25232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:22.025761 25231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:22.035337 25250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:22.041306 25250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:22.051577 25250 catalog_manager.cc:1347] Generated new cluster ID: a833048f1be648c9bd932539bfd72e21
I20260808 13:02:22.051731 25250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:22.073921 25250 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:22.075971 25250 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:22.093250 25250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aa9175577904d98910dc22b0f1c6348: Generated new TSK 0
I20260808 13:02:22.094534 25250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:22.108286 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 25120
W20260808 13:02:22.109479 25207 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.86:38143
W20260808 13:02:22.110479 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 25120: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 13:02:22.110648 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 25120
W20260808 13:02:22.138101 25102 connection.cc:535] client connection to 127.0.0.1:39345 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39345): Connection reset by peer (error 104)
2026-08-08T13:02:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:22 dist-test-slave-tk9z krb5kdc[25266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:22 dist-test-slave-tk9z krb5kdc[25266](info): set up 2 sockets
Aug 08 13:02:22 dist-test-slave-tk9z krb5kdc[25266](info): commencing operation
krb5kdc: starting...
W20260808 13:02:23.473546 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.316s user 0.009s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:23 dist-test-slave-tk9z krb5kdc[25266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 13:02:23.806182 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:24.158469 25279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:24.158779 25279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:24.158887 25279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:24.166124 25279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:24.166234 25279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:24.166292 25279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:24.166355 25279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:24.166481 25279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:24.176518 25279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:24.177827 25279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:24.179230 25279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:24.191483 25284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:24.191533 25285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:24.191561 25287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:24.192734 25279 server_base.cc:1034] running on GCE node
I20260808 13:02:24.193511 25279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:24.194938 25279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:24.196127 25279 hybrid_clock.cc:648] HybridClock initialized: now 1786194144196067 us; error 51 us; skew 500 ppm
I20260808 13:02:24.196424 25279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:24 dist-test-slave-tk9z krb5kdc[25266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:24.202773 25279 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 13:02:24.204301 25279 webserver.cc:460] Webserver started at http://127.0.0.1:39243/ using document root <none> and password file <none>
I20260808 13:02:24.204952 25279 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:24.205055 25279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:24.205307 25279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:24.208333 25279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "253d8a964aee41bb8532a7564c6b5054"
format_stamp: "Formatted at 2026-08-08 13:02:24 on dist-test-slave-tk9z"
server_key: "34d0ed87a1013b67b7d29f2f80c5426a"
server_key_iv: "f4ed24a11342653a336aa700baa611be"
server_key_version: "encryptionkey@0"
I20260808 13:02:24.209057 25279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "253d8a964aee41bb8532a7564c6b5054"
format_stamp: "Formatted at 2026-08-08 13:02:24 on dist-test-slave-tk9z"
server_key: "34d0ed87a1013b67b7d29f2f80c5426a"
server_key_iv: "f4ed24a11342653a336aa700baa611be"
server_key_version: "encryptionkey@0"
I20260808 13:02:24.214118 25279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 13:02:24.217696 25294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:24.219143 25279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:02:24.219296 25279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "253d8a964aee41bb8532a7564c6b5054"
format_stamp: "Formatted at 2026-08-08 13:02:24 on dist-test-slave-tk9z"
server_key: "34d0ed87a1013b67b7d29f2f80c5426a"
server_key_iv: "f4ed24a11342653a336aa700baa611be"
server_key_version: "encryptionkey@0"
I20260808 13:02:24.219488 25279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:24.225556 25279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:24.229995 25279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:24.230257 25279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:24.249152 25279 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36585
I20260808 13:02:24.249198 25364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36585 every 8 connection(s)
I20260808 13:02:24.250634 25279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:24.254357 25366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:24.257876 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 25279
I20260808 13:02:24.258085 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:24.258613 25244 external_mini_cluster.cc:1428] Setting key 1efac7ad8b2b114d9df8b505aaef6840
I20260808 13:02:24.267480 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:24.260504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:47183 (local address 127.0.0.1:36585)
0808 13:02:24.261332 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:24.261348 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:24.261402 (+ 54us) server_negotiation.cc:407] Connection header received
0808 13:02:24.263490 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:24.263521 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:24.263992 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:24.264442 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:24.265522 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:24.265535 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:24.265552 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:24.265578 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:24.265655 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:24.265686 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:24.265917 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:24.266071 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 13:02:24.266269 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":324,"threads_started":1}
I20260808 13:02:24.268262 25366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054: Bootstrap starting.
I20260808 13:02:24.271433 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:24.272064 25366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:24.273063 25366 log.cc:824] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:24.276100 25366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054: No bootstrap required, opened a new log
I20260808 13:02:24.278766 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:24.276096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:40103 (local address 127.0.0.1:36585)
0808 13:02:24.276262 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:24.276270 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:24.276712 (+ 442us) server_negotiation.cc:407] Connection header received
0808 13:02:24.276906 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:24.276913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:24.276964 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:24.277067 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:24.278044 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:24.278056 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:24.278059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:24.278074 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:24.278113 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:24.278129 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:24.278258 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:24.278439 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 13:02:24.278504 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 13:02:24.284716 25366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253d8a964aee41bb8532a7564c6b5054" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } }
I20260808 13:02:24.285090 25366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:24.285202 25366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253d8a964aee41bb8532a7564c6b5054, State: Initialized, Role: FOLLOWER
I20260808 13:02:24.285799 25366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253d8a964aee41bb8532a7564c6b5054" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } }
I20260808 13:02:24.286015 25366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:24.286199 25366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:24.286360 25366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:24.289014 25366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253d8a964aee41bb8532a7564c6b5054" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } }
I20260808 13:02:24.289526 25366 leader_election.cc:302] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 253d8a964aee41bb8532a7564c6b5054; no voters:
I20260808 13:02:24.290100 25366 leader_election.cc:288] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:24.290241 25379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:24.291004 25379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [term 1 LEADER]: Becoming Leader. State: Replica: 253d8a964aee41bb8532a7564c6b5054, State: Running, Role: LEADER
I20260808 13:02:24.291509 25379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253d8a964aee41bb8532a7564c6b5054" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } }
I20260808 13:02:24.292429 25366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:24.294580 25381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 253d8a964aee41bb8532a7564c6b5054. Latest consensus state: current_term: 1 leader_uuid: "253d8a964aee41bb8532a7564c6b5054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253d8a964aee41bb8532a7564c6b5054" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } } }
I20260808 13:02:24.294610 25380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "253d8a964aee41bb8532a7564c6b5054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253d8a964aee41bb8532a7564c6b5054" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } } }
I20260808 13:02:24.295084 25381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:24.295084 25380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:24.297194 25387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:24.302471 25387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:24.313022 25387 catalog_manager.cc:1347] Generated new cluster ID: d60b8861b6164916b7858bc73b780562
I20260808 13:02:24.313182 25387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:24.340890 25387 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:24.341379 25387 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:24.346796 25387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 253d8a964aee41bb8532a7564c6b5054: Generated new TSK 0
I20260808 13:02:24.347774 25387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:24.353324 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 25279
W20260808 13:02:24.354598 25360 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.86:47183
W20260808 13:02:24.355432 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 25279: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 13:02:24.355614 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 25279
2026-08-08T13:02:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2247 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:24 dist-test-slave-tk9z krb5kdc[25411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:24 dist-test-slave-tk9z krb5kdc[25411](info): set up 2 sockets
Aug 08 13:02:24 dist-test-slave-tk9z krb5kdc[25411](info): commencing operation
krb5kdc: starting...
W20260808 13:02:25.681360 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:25 dist-test-slave-tk9z krb5kdc[25411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 13:02:26.013494 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:26.369234 25433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:26.369619 25433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:26.369807 25433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:26.377070 25433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:26.377188 25433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:26.377310 25433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:26.377393 25433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:26.377506 25433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:26.387723 25433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:26.389185 25433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:26.390654 25433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:26.403564 25439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:26.403650 25443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:26.403687 25440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:26.404718 25433 server_base.cc:1034] running on GCE node
I20260808 13:02:26.405689 25433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:26.407371 25433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:26.408651 25433 hybrid_clock.cc:648] HybridClock initialized: now 1786194146408608 us; error 51 us; skew 500 ppm
I20260808 13:02:26.409056 25433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:26 dist-test-slave-tk9z krb5kdc[25411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:26.412510 25433 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 13:02:26.414204 25433 webserver.cc:460] Webserver started at http://127.0.0.1:43421/ using document root <none> and password file <none>
I20260808 13:02:26.414968 25433 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:26.415117 25433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:26.415444 25433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:26.418628 25433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "9d05ea1e6e7d4598b20d8da177302f7f"
format_stamp: "Formatted at 2026-08-08 13:02:26 on dist-test-slave-tk9z"
server_key: "8b3d396f67fc8f5288df8ecc59d689f4"
server_key_iv: "2632bd95a2363cce4e982ae35df08e3a"
server_key_version: "encryptionkey@0"
I20260808 13:02:26.419464 25433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "9d05ea1e6e7d4598b20d8da177302f7f"
format_stamp: "Formatted at 2026-08-08 13:02:26 on dist-test-slave-tk9z"
server_key: "8b3d396f67fc8f5288df8ecc59d689f4"
server_key_iv: "2632bd95a2363cce4e982ae35df08e3a"
server_key_version: "encryptionkey@0"
I20260808 13:02:26.424882 25433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 13:02:26.428570 25450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:26.430090 25433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 13:02:26.430301 25433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "9d05ea1e6e7d4598b20d8da177302f7f"
format_stamp: "Formatted at 2026-08-08 13:02:26 on dist-test-slave-tk9z"
server_key: "8b3d396f67fc8f5288df8ecc59d689f4"
server_key_iv: "2632bd95a2363cce4e982ae35df08e3a"
server_key_version: "encryptionkey@0"
I20260808 13:02:26.430588 25433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:26.445277 25433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:26.449369 25433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:26.449683 25433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:26.469381 25433 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45857
I20260808 13:02:26.469415 25520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45857 every 8 connection(s)
I20260808 13:02:26.471028 25433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:26.474977 25522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:26.477378 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 25433
I20260808 13:02:26.477572 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:26.478140 25244 external_mini_cluster.cc:1428] Setting key a11713454dd6a578a2f5a4e673fca3de
I20260808 13:02:26.487998 25522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f: Bootstrap starting.
I20260808 13:02:26.492182 25522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f: Neither blocks nor log segments found. Creating new log.
Aug 08 13:02:26 dist-test-slave-tk9z krb5kdc[25411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 13:02:26.493472 25522 log.cc:824] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:26.496567 25522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f: No bootstrap required, opened a new log
I20260808 13:02:26.502223 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:26.480050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:40633 (local address 127.0.0.1:45857)
0808 13:02:26.480868 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:26.480879 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:26.480924 (+ 45us) server_negotiation.cc:407] Connection header received
0808 13:02:26.482667 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:26.482690 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:26.483064 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:26.483359 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:26.485251 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:26.486660 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:26.488531 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:26.489198 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:26.489339 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:26.494567 (+ 5228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:26.494576 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:26.494591 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:26.494632 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:26.497802 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:26.498695 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:26.498701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:26.498707 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:26.498886 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:26.499387 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:26.499390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:26.499392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:26.499951 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:26.500294 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:26.500865 (+ 571us) server_negotiation.cc:299] Negotiation successful
0808 13:02:26.501202 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":379,"threads_started":1}
I20260808 13:02:26.504859 25522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d05ea1e6e7d4598b20d8da177302f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45857 } }
I20260808 13:02:26.505337 25522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:26.505453 25522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d05ea1e6e7d4598b20d8da177302f7f, State: Initialized, Role: FOLLOWER
I20260808 13:02:26.506057 25522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d05ea1e6e7d4598b20d8da177302f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45857 } }
I20260808 13:02:26.506263 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:26.506363 25522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:26.506569 25522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:26.506749 25522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:26.509030 25522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d05ea1e6e7d4598b20d8da177302f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45857 } }
I20260808 13:02:26.509610 25522 leader_election.cc:302] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d05ea1e6e7d4598b20d8da177302f7f; no voters:
I20260808 13:02:26.510283 25522 leader_election.cc:288] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:26.510427 25536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:26.514483 25536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [term 1 LEADER]: Becoming Leader. State: Replica: 9d05ea1e6e7d4598b20d8da177302f7f, State: Running, Role: LEADER
I20260808 13:02:26.515260 25536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d05ea1e6e7d4598b20d8da177302f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45857 } }
I20260808 13:02:26.519337 25539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d05ea1e6e7d4598b20d8da177302f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d05ea1e6e7d4598b20d8da177302f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45857 } } }
I20260808 13:02:26.519666 25522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:26.519780 25539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:26.520181 25537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d05ea1e6e7d4598b20d8da177302f7f. Latest consensus state: current_term: 1 leader_uuid: "9d05ea1e6e7d4598b20d8da177302f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d05ea1e6e7d4598b20d8da177302f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45857 } } }
I20260808 13:02:26.520532 25537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:26.527125 25543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:26.528767 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:26.510894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:57045 (local address 127.0.0.1:45857)
0808 13:02:26.511113 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:26.511122 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:26.511653 (+ 531us) server_negotiation.cc:407] Connection header received
0808 13:02:26.511883 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:26.511889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:26.511960 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:26.512056 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:26.513501 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:26.514312 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:26.515736 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:26.516210 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:26.516348 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:26.519764 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:26.519771 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:26.519778 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:26.519826 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:26.524440 (+ 4614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:26.525743 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:26.525748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:26.525752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:26.525898 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:26.526575 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:26.526585 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:26.526588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:26.527184 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:26.527518 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:26.528147 (+ 629us) server_negotiation.cc:299] Negotiation successful
0808 13:02:26.528352 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 13:02:26.536623 25543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:26.545594 25543 catalog_manager.cc:1347] Generated new cluster ID: 55694e54fb044b28989e86ca6f1a9d5a
I20260808 13:02:26.545713 25543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:26.575055 25543 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:26.576999 25543 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:26.587654 25543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d05ea1e6e7d4598b20d8da177302f7f: Generated new TSK 0
I20260808 13:02:26.588725 25543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:26.597373 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 25433
I20260808 13:02:26.598634 25515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:26.691643 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 25433
W20260808 13:02:26.718184 25407 connection.cc:535] client connection to 127.0.0.1:45857 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45857): Connection reset by peer (error 104)
2026-08-08T13:02:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:26 dist-test-slave-tk9z krb5kdc[25567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:26 dist-test-slave-tk9z krb5kdc[25567](info): set up 2 sockets
Aug 08 13:02:26 dist-test-slave-tk9z krb5kdc[25567](info): commencing operation
krb5kdc: starting...
W20260808 13:02:28.031365 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.289s user 0.012s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:28 dist-test-slave-tk9z krb5kdc[25567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 13:02:28.354542 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:28.710861 25585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:28.711181 25585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:28.711278 25585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:28.718783 25585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:28.718917 25585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:28.718973 25585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:28.719054 25585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:28.719100 25585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:28.729776 25585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34473
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:28.731305 25585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:28.732625 25585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:28.744951 25592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:28.744969 25591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:28.745007 25595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:28.746294 25585 server_base.cc:1034] running on GCE node
I20260808 13:02:28.747151 25585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:28.748600 25585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:28.749801 25585 hybrid_clock.cc:648] HybridClock initialized: now 1786194148749739 us; error 52 us; skew 500 ppm
I20260808 13:02:28.750180 25585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:28 dist-test-slave-tk9z krb5kdc[25567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:28.753540 25585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 13:02:28.755198 25585 webserver.cc:460] Webserver started at http://127.0.0.1:44501/ using document root <none> and password file <none>
I20260808 13:02:28.755923 25585 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:28.756084 25585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:28.756408 25585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:28.759562 25585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "5a48187e491a43d3812d8d481aa42ef3"
format_stamp: "Formatted at 2026-08-08 13:02:28 on dist-test-slave-tk9z"
server_key: "5806a97225b1e46d128d360d48cd497d"
server_key_iv: "35e0371ce7dfe2e14ac6efb0d3ef447e"
server_key_version: "encryptionkey@0"
I20260808 13:02:28.760361 25585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "5a48187e491a43d3812d8d481aa42ef3"
format_stamp: "Formatted at 2026-08-08 13:02:28 on dist-test-slave-tk9z"
server_key: "5806a97225b1e46d128d360d48cd497d"
server_key_iv: "35e0371ce7dfe2e14ac6efb0d3ef447e"
server_key_version: "encryptionkey@0"
I20260808 13:02:28.765748 25585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 13:02:28.769289 25604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:28.770776 25585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:02:28.770973 25585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "5a48187e491a43d3812d8d481aa42ef3"
format_stamp: "Formatted at 2026-08-08 13:02:28 on dist-test-slave-tk9z"
server_key: "5806a97225b1e46d128d360d48cd497d"
server_key_iv: "35e0371ce7dfe2e14ac6efb0d3ef447e"
server_key_version: "encryptionkey@0"
I20260808 13:02:28.771180 25585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:28.788023 25585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:28.792531 25585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:28.792836 25585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:28.812160 25585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34473
I20260808 13:02:28.812199 25678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34473 every 8 connection(s)
I20260808 13:02:28.813611 25585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:28.815338 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 25585
I20260808 13:02:28.815532 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:28.816156 25244 external_mini_cluster.cc:1428] Setting key 722c83580f9bce4738a71c2762e76357
I20260808 13:02:28.818545 25679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:28.829702 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:28.818027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:36477 (local address 127.0.0.1:34473)
0808 13:02:28.818911 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:28.818924 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:28.818965 (+ 41us) server_negotiation.cc:407] Connection header received
0808 13:02:28.820582 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:28.820604 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:28.820984 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:28.821316 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:28.821918 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:28.823340 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:28.825242 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:28.825904 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:28.826032 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:28.827382 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:28.827389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:28.827432 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:28.827974 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:28.828049 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:28.828074 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:28.828200 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:28.828562 (+ 362us) server_negotiation.cc:299] Negotiation successful
0808 13:02:28.828758 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":454,"threads_started":1}
I20260808 13:02:28.831314 25679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3: Bootstrap starting.
I20260808 13:02:28.833726 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:28.834857 25679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:28.835855 25679 log.cc:824] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:28.838817 25679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3: No bootstrap required, opened a new log
I20260808 13:02:28.844585 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:28.838226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:55557 (local address 127.0.0.1:34473)
0808 13:02:28.838430 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:28.838436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:28.838934 (+ 498us) server_negotiation.cc:407] Connection header received
0808 13:02:28.839140 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:28.839144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:28.839205 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:28.839318 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:28.839822 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:28.840602 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:28.841959 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:28.842456 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:28.842624 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:28.843797 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:28.843801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:28.843805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:28.843821 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:28.843874 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:28.843892 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:28.844039 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:28.844240 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 13:02:28.844300 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 13:02:28.847229 25679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a48187e491a43d3812d8d481aa42ef3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } }
I20260808 13:02:28.847632 25679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:28.847760 25679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a48187e491a43d3812d8d481aa42ef3, State: Initialized, Role: FOLLOWER
I20260808 13:02:28.848356 25679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a48187e491a43d3812d8d481aa42ef3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } }
I20260808 13:02:28.848628 25679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:28.848779 25679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:28.848986 25679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:28.851686 25679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a48187e491a43d3812d8d481aa42ef3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } }
I20260808 13:02:28.852192 25679 leader_election.cc:302] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a48187e491a43d3812d8d481aa42ef3; no voters:
I20260808 13:02:28.852787 25679 leader_election.cc:288] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:28.852936 25690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:28.853720 25690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [term 1 LEADER]: Becoming Leader. State: Replica: 5a48187e491a43d3812d8d481aa42ef3, State: Running, Role: LEADER
I20260808 13:02:28.854322 25690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a48187e491a43d3812d8d481aa42ef3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } }
I20260808 13:02:28.855074 25679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:28.857478 25692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a48187e491a43d3812d8d481aa42ef3. Latest consensus state: current_term: 1 leader_uuid: "5a48187e491a43d3812d8d481aa42ef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a48187e491a43d3812d8d481aa42ef3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } } }
I20260808 13:02:28.857869 25692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:28.857578 25691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a48187e491a43d3812d8d481aa42ef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a48187e491a43d3812d8d481aa42ef3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } } }
I20260808 13:02:28.858031 25691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:28.860570 25697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:28.865599 25697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:28.874148 25697 catalog_manager.cc:1347] Generated new cluster ID: 47f660fbfc1d49c8aa476f0dfc9c6b1f
I20260808 13:02:28.874274 25697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:28.882596 25697 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:28.884469 25697 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:28.901525 25697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a48187e491a43d3812d8d481aa42ef3: Generated new TSK 0
I20260808 13:02:28.902722 25697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:28.917191 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 25585
W20260808 13:02:28.919051 25674 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.86:36477
W20260808 13:02:28.919836 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 25585: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 13:02:28.919983 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 25585
W20260808 13:02:28.942107 25563 connection.cc:535] client connection to 127.0.0.1:34473 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34473): Connection reset by peer (error 104)
2026-08-08T13:02:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:29 dist-test-slave-tk9z krb5kdc[25719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:29 dist-test-slave-tk9z krb5kdc[25719](info): set up 2 sockets
Aug 08 13:02:29 dist-test-slave-tk9z krb5kdc[25719](info): commencing operation
krb5kdc: starting...
W20260808 13:02:30.266194 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.305s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:30 dist-test-slave-tk9z krb5kdc[25719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 13:02:30.591857 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:30.953653 25738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:30.953953 25738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:30.954048 25738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:30.962005 25738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:30.962126 25738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:30.962174 25738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:30.962237 25738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:30.962281 25738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:30.972389 25738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:30.973768 25738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:30.975112 25738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:30.987609 25745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:30.987576 25749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:30.987520 25746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:30.988555 25738 server_base.cc:1034] running on GCE node
I20260808 13:02:30.989394 25738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:30.990902 25738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:30.992141 25738 hybrid_clock.cc:648] HybridClock initialized: now 1786194150992101 us; error 36 us; skew 500 ppm
I20260808 13:02:30.992540 25738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:30 dist-test-slave-tk9z krb5kdc[25719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:30.999289 25738 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 13:02:31.000946 25738 webserver.cc:460] Webserver started at http://127.0.0.1:38849/ using document root <none> and password file <none>
I20260808 13:02:31.001655 25738 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:31.001765 25738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:31.002043 25738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:31.005472 25738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "f237c74382ea4b46802173ec54bf2d99"
format_stamp: "Formatted at 2026-08-08 13:02:31 on dist-test-slave-tk9z"
server_key: "6bf53c7f1a8883b613b00a0f32246ded"
server_key_iv: "bacc44516e7817f5bc901f01e1f1bc2d"
server_key_version: "encryptionkey@0"
I20260808 13:02:31.006242 25738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "f237c74382ea4b46802173ec54bf2d99"
format_stamp: "Formatted at 2026-08-08 13:02:31 on dist-test-slave-tk9z"
server_key: "6bf53c7f1a8883b613b00a0f32246ded"
server_key_iv: "bacc44516e7817f5bc901f01e1f1bc2d"
server_key_version: "encryptionkey@0"
I20260808 13:02:31.011765 25738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260808 13:02:31.015290 25757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:31.016662 25738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 13:02:31.016858 25738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "f237c74382ea4b46802173ec54bf2d99"
format_stamp: "Formatted at 2026-08-08 13:02:31 on dist-test-slave-tk9z"
server_key: "6bf53c7f1a8883b613b00a0f32246ded"
server_key_iv: "bacc44516e7817f5bc901f01e1f1bc2d"
server_key_version: "encryptionkey@0"
I20260808 13:02:31.017088 25738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:31.023250 25738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:31.027431 25738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:31.027743 25738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:31.046772 25738 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34353
I20260808 13:02:31.046818 25825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34353 every 8 connection(s)
I20260808 13:02:31.048271 25738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:31.052318 25827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:31.054414 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 25738
I20260808 13:02:31.054610 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:31.055058 25244 external_mini_cluster.cc:1428] Setting key 41df165530a2a99c399a2025180e47c7
I20260808 13:02:31.063153 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:31.056992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:54861 (local address 127.0.0.1:34353)
0808 13:02:31.057807 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:31.057817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:31.057858 (+ 41us) server_negotiation.cc:407] Connection header received
0808 13:02:31.059598 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:31.059621 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:31.059959 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:31.060261 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:31.061427 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:31.061436 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:31.061448 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:31.061462 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:31.061524 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:31.061548 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:31.061747 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:31.061872 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 13:02:31.062142 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":391,"threads_started":1}
I20260808 13:02:31.064935 25827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99: Bootstrap starting.
I20260808 13:02:31.066982 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:31.068404 25827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:31.069445 25827 log.cc:824] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:31.072840 25827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99: No bootstrap required, opened a new log
I20260808 13:02:31.074440 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:31.071415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.86:38493 (local address 127.0.0.1:34353)
0808 13:02:31.071621 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:31.071629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:31.072164 (+ 535us) server_negotiation.cc:407] Connection header received
0808 13:02:31.072345 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:31.072349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:31.072395 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:31.072481 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:31.073531 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:31.073536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:31.073543 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:31.073559 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:31.073601 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:31.073616 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:31.073712 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:31.073929 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 13:02:31.074026 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 13:02:31.080653 25827 raft_consensus.cc:348] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f237c74382ea4b46802173ec54bf2d99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34353 } }
I20260808 13:02:31.081074 25827 raft_consensus.cc:374] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:31.081199 25827 raft_consensus.cc:729] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f237c74382ea4b46802173ec54bf2d99, State: Initialized, Role: FOLLOWER
I20260808 13:02:31.081807 25827 consensus_queue.cc:260] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f237c74382ea4b46802173ec54bf2d99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34353 } }
I20260808 13:02:31.082057 25827 raft_consensus.cc:388] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:31.082216 25827 raft_consensus.cc:482] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:31.082432 25827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:31.084689 25827 raft_consensus.cc:504] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f237c74382ea4b46802173ec54bf2d99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34353 } }
I20260808 13:02:31.085194 25827 leader_election.cc:302] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f237c74382ea4b46802173ec54bf2d99; no voters:
I20260808 13:02:31.085794 25827 leader_election.cc:288] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:31.085954 25840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:31.086663 25840 raft_consensus.cc:686] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [term 1 LEADER]: Becoming Leader. State: Replica: f237c74382ea4b46802173ec54bf2d99, State: Running, Role: LEADER
I20260808 13:02:31.087301 25840 consensus_queue.cc:237] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f237c74382ea4b46802173ec54bf2d99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34353 } }
I20260808 13:02:31.088003 25827 sys_catalog.cc:564] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:31.090512 25842 sys_catalog.cc:455] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f237c74382ea4b46802173ec54bf2d99. Latest consensus state: current_term: 1 leader_uuid: "f237c74382ea4b46802173ec54bf2d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f237c74382ea4b46802173ec54bf2d99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34353 } } }
I20260808 13:02:31.090631 25841 sys_catalog.cc:455] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f237c74382ea4b46802173ec54bf2d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f237c74382ea4b46802173ec54bf2d99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34353 } } }
I20260808 13:02:31.090991 25842 sys_catalog.cc:458] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:31.091158 25841 sys_catalog.cc:458] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:31.093603 25854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:31.098392 25854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:31.108068 25854 catalog_manager.cc:1347] Generated new cluster ID: 27e6fa3bac944ce8bc5220b89c8818af
I20260808 13:02:31.108192 25854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:31.121464 25854 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:31.121984 25854 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:31.137208 25854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f237c74382ea4b46802173ec54bf2d99: Generated new TSK 0
I20260808 13:02:31.138414 25854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:31.151144 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 25738
W20260808 13:02:31.152418 25821 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.86:54861
W20260808 13:02:31.153200 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 25738: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 13:02:31.153355 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 25738
2026-08-08T13:02:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2237 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:31 dist-test-slave-tk9z krb5kdc[25870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:31 dist-test-slave-tk9z krb5kdc[25870](info): set up 2 sockets
Aug 08 13:02:31 dist-test-slave-tk9z krb5kdc[25870](info): commencing operation
krb5kdc: starting...
W20260808 13:02:32.565855 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.367s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:32 dist-test-slave-tk9z krb5kdc[25870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 13:02:32.893769 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:34617
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:34617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:33.273532 25884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:33.273844 25884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:33.274025 25884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:33.281386 25884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:33.281509 25884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:33.281651 25884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:33.281759 25884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:33.281842 25884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:33.292198 25884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:34617
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:34617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:33.293717 25884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:33.295186 25884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:33.307804 25889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:33.307847 25892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:33.307761 25890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:33.308789 25884 server_base.cc:1034] running on GCE node
I20260808 13:02:33.309690 25884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:33.311301 25884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:33.312543 25884 hybrid_clock.cc:648] HybridClock initialized: now 1786194153312502 us; error 58 us; skew 500 ppm
I20260808 13:02:33.312933 25884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:33 dist-test-slave-tk9z krb5kdc[25870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194153, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:33.316260 25884 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:02:33.317744 25884 webserver.cc:460] Webserver started at http://127.24.167.62:36309/ using document root <none> and password file <none>
I20260808 13:02:33.318480 25884 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:33.318593 25884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:33.318849 25884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:33.321857 25884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "5e89a96f01684b2f8a805cc29f4abeb6"
format_stamp: "Formatted at 2026-08-08 13:02:33 on dist-test-slave-tk9z"
server_key: "ae9d4c597750aa6637c7083324f5b8ab"
server_key_iv: "30c42278c1745eba21a046b2dd278319"
server_key_version: "encryptionkey@0"
I20260808 13:02:33.322686 25884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "5e89a96f01684b2f8a805cc29f4abeb6"
format_stamp: "Formatted at 2026-08-08 13:02:33 on dist-test-slave-tk9z"
server_key: "ae9d4c597750aa6637c7083324f5b8ab"
server_key_iv: "30c42278c1745eba21a046b2dd278319"
server_key_version: "encryptionkey@0"
I20260808 13:02:33.327847 25884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260808 13:02:33.331229 25902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:33.332551 25884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 13:02:33.332755 25884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "5e89a96f01684b2f8a805cc29f4abeb6"
format_stamp: "Formatted at 2026-08-08 13:02:33 on dist-test-slave-tk9z"
server_key: "ae9d4c597750aa6637c7083324f5b8ab"
server_key_iv: "30c42278c1745eba21a046b2dd278319"
server_key_version: "encryptionkey@0"
I20260808 13:02:33.332962 25884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:33.351261 25884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:33.355935 25884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:33.356253 25884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:33.375720 25884 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:34617
I20260808 13:02:33.375762 25973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:34617 every 8 connection(s)
I20260808 13:02:33.377137 25884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:33.381348 25974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:33.384389 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 25884
I20260808 13:02:33.384619 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:33.385231 25244 external_mini_cluster.cc:1428] Setting key 84b766735d7a804c1ded22190edf9281
I20260808 13:02:33.395159 25974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6: Bootstrap starting.
I20260808 13:02:33.398653 25974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:33.399780 25974 log.cc:824] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6: Log is configured to *not* fsync() on all Append() calls
Aug 08 13:02:33 dist-test-slave-tk9z krb5kdc[25870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:02:33.403760 25974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6: No bootstrap required, opened a new log
I20260808 13:02:33.411036 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:33.387485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48421 (local address 127.24.167.62:34617)
0808 13:02:33.388468 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:33.388481 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:33.388530 (+ 49us) server_negotiation.cc:407] Connection header received
0808 13:02:33.390238 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:33.390274 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:33.390773 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:33.391177 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:33.392781 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:33.394150 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:33.395588 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:33.396287 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:33.396489 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:33.401321 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:33.401334 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:33.401356 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:33.401406 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:33.405697 (+ 4291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:33.406818 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:33.406827 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:33.406841 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:33.407096 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:33.407572 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:33.407577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:33.407581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:33.408381 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:33.408741 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:33.409117 (+ 376us) server_negotiation.cc:299] Negotiation successful
0808 13:02:33.409508 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":482,"threads_started":1}
I20260808 13:02:33.414849 25974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89a96f01684b2f8a805cc29f4abeb6" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34617 } }
I20260808 13:02:33.415338 25974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:33.415418 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:33.415486 25974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e89a96f01684b2f8a805cc29f4abeb6, State: Initialized, Role: FOLLOWER
I20260808 13:02:33.416134 25974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89a96f01684b2f8a805cc29f4abeb6" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34617 } }
I20260808 13:02:33.416435 25974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:33.416594 25974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:33.416765 25974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:33.420385 25974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89a96f01684b2f8a805cc29f4abeb6" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34617 } }
I20260808 13:02:33.421077 25974 leader_election.cc:302] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e89a96f01684b2f8a805cc29f4abeb6; no voters:
I20260808 13:02:33.421912 25974 leader_election.cc:288] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:33.422087 25985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:33.423475 25985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [term 1 LEADER]: Becoming Leader. State: Replica: 5e89a96f01684b2f8a805cc29f4abeb6, State: Running, Role: LEADER
I20260808 13:02:33.424391 25974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:33.424266 25985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89a96f01684b2f8a805cc29f4abeb6" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34617 } }
I20260808 13:02:33.428261 25986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e89a96f01684b2f8a805cc29f4abeb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89a96f01684b2f8a805cc29f4abeb6" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34617 } } }
I20260808 13:02:33.428700 25986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:33.429260 25987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e89a96f01684b2f8a805cc29f4abeb6. Latest consensus state: current_term: 1 leader_uuid: "5e89a96f01684b2f8a805cc29f4abeb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89a96f01684b2f8a805cc29f4abeb6" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34617 } } }
I20260808 13:02:33.429579 25987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:33.436851 25996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:33.441748 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:33.421145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60523 (local address 127.24.167.62:34617)
0808 13:02:33.421384 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:33.421396 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:33.421974 (+ 578us) server_negotiation.cc:407] Connection header received
0808 13:02:33.422175 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:33.422188 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:33.422287 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:33.424565 (+ 2278us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:33.424924 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:33.426361 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:33.428509 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:33.429315 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:33.429494 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:33.432760 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:33.432770 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:33.432778 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:33.432833 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:33.438041 (+ 5208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:33.439112 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:33.439117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:33.439122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:33.439281 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:33.439822 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:33.439829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:33.439835 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:33.440311 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:33.440585 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:33.441070 (+ 485us) server_negotiation.cc:299] Negotiation successful
0808 13:02:33.441275 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":78}
I20260808 13:02:33.442983 25996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:33.454921 25996 catalog_manager.cc:1347] Generated new cluster ID: 7bda97f6d01d40d9861ab3d09932422e
I20260808 13:02:33.455075 25996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:33.470681 25996 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:33.472651 25996 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:33.486567 25996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e89a96f01684b2f8a805cc29f4abeb6: Generated new TSK 0
I20260808 13:02:33.487664 25996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:33.500778 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 25884
I20260808 13:02:33.502255 25966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:33.599004 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 25884
W20260808 13:02:33.624773 25867 connection.cc:535] client connection to 127.24.167.62:34617 recv error: Network error: failed to read from TLS socket (remote: 127.24.167.62:34617): Connection reset by peer (error 104)
2026-08-08T13:02:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:33 dist-test-slave-tk9z krb5kdc[26019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:33 dist-test-slave-tk9z krb5kdc[26019](info): set up 2 sockets
Aug 08 13:02:33 dist-test-slave-tk9z krb5kdc[26019](info): commencing operation
krb5kdc: starting...
W20260808 13:02:34.949611 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.305s user 0.004s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:35 dist-test-slave-tk9z krb5kdc[26019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 13:02:35.287853 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:35907
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:35907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:35.655296 26038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:35.655642 26038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:35.655866 26038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:35.664285 26038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:35.664431 26038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:35.664542 26038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:35.664637 26038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:35.664732 26038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:35.676079 26038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:35907
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:35907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:35.677623 26038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:35.679150 26038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:35.692507 26046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:35.692605 26050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:35.692582 26047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:35.693771 26038 server_base.cc:1034] running on GCE node
I20260808 13:02:35.694716 26038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:35.696228 26038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:35.697479 26038 hybrid_clock.cc:648] HybridClock initialized: now 1786194155697434 us; error 52 us; skew 500 ppm
I20260808 13:02:35.697849 26038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:35 dist-test-slave-tk9z krb5kdc[26019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194155, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:35.701186 26038 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:02:35.702904 26038 webserver.cc:460] Webserver started at http://127.24.167.62:43085/ using document root <none> and password file <none>
I20260808 13:02:35.703650 26038 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:35.703805 26038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:35.704133 26038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:35.707301 26038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "7385fa92d113499889eb4c2aaf270aec"
format_stamp: "Formatted at 2026-08-08 13:02:35 on dist-test-slave-tk9z"
server_key: "32aedefb273023f2534e125ab820fe7b"
server_key_iv: "441db94808c955e7137a1d97455ff601"
server_key_version: "encryptionkey@0"
I20260808 13:02:35.708118 26038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "7385fa92d113499889eb4c2aaf270aec"
format_stamp: "Formatted at 2026-08-08 13:02:35 on dist-test-slave-tk9z"
server_key: "32aedefb273023f2534e125ab820fe7b"
server_key_iv: "441db94808c955e7137a1d97455ff601"
server_key_version: "encryptionkey@0"
I20260808 13:02:35.713559 26038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 13:02:35.717170 26058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:35.718561 26038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:02:35.718757 26038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "7385fa92d113499889eb4c2aaf270aec"
format_stamp: "Formatted at 2026-08-08 13:02:35 on dist-test-slave-tk9z"
server_key: "32aedefb273023f2534e125ab820fe7b"
server_key_iv: "441db94808c955e7137a1d97455ff601"
server_key_version: "encryptionkey@0"
I20260808 13:02:35.719002 26038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:35.731298 26038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:35.735256 26038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:35.735567 26038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:35.755092 26038 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:35907
I20260808 13:02:35.755110 26131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:35907 every 8 connection(s)
I20260808 13:02:35.756595 26038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:35.760636 26133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:35.760757 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 26038
I20260808 13:02:35.760958 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:35.761512 25244 external_mini_cluster.cc:1428] Setting key 1884f4d10d1a09d879643870920ad451
I20260808 13:02:35.776825 26136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:35.763531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56331 (local address 127.24.167.62:35907)
0808 13:02:35.764544 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:35.764556 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:35.764600 (+ 44us) server_negotiation.cc:407] Connection header received
0808 13:02:35.766322 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:35.766356 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:35.767048 (+ 692us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:35.767486 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:35.768258 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:35.769816 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:35.771823 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:35.772552 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:35.772709 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:35.774272 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:35.774280 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:35.774299 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:35.774976 (+ 677us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:35.775059 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:35.775089 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:35.775218 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:35.775562 (+ 344us) server_negotiation.cc:299] Negotiation successful
0808 13:02:35.775779 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":456,"threads_started":1}
I20260808 13:02:35.776870 26133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec: Bootstrap starting.
I20260808 13:02:35.780365 26133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:35.780993 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:35.781692 26133 log.cc:824] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:35.785226 26133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec: No bootstrap required, opened a new log
I20260808 13:02:35.793308 26136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:35.786136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43529 (local address 127.24.167.62:35907)
0808 13:02:35.786322 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:35.786328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:35.786785 (+ 457us) server_negotiation.cc:407] Connection header received
0808 13:02:35.786950 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:35.786960 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:35.787047 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:35.787164 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:35.787650 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:35.788844 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:35.790253 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:35.791043 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:35.791185 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:35.792463 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:35.792470 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:35.792475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:35.792499 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:35.792556 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:35.792577 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:35.792715 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:35.792879 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 13:02:35.792954 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 13:02:35.795272 26133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7385fa92d113499889eb4c2aaf270aec" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35907 } }
I20260808 13:02:35.795743 26133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:35.795876 26133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7385fa92d113499889eb4c2aaf270aec, State: Initialized, Role: FOLLOWER
I20260808 13:02:35.796494 26133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7385fa92d113499889eb4c2aaf270aec" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35907 } }
I20260808 13:02:35.796746 26133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:35.796911 26133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:35.797114 26133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:35.799507 26133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7385fa92d113499889eb4c2aaf270aec" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35907 } }
I20260808 13:02:35.800086 26133 leader_election.cc:302] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7385fa92d113499889eb4c2aaf270aec; no voters:
I20260808 13:02:35.800738 26133 leader_election.cc:288] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:35.800894 26143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:35.802201 26143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [term 1 LEADER]: Becoming Leader. State: Replica: 7385fa92d113499889eb4c2aaf270aec, State: Running, Role: LEADER
I20260808 13:02:35.802753 26143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7385fa92d113499889eb4c2aaf270aec" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35907 } }
I20260808 13:02:35.803169 26133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:35.805996 26144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7385fa92d113499889eb4c2aaf270aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7385fa92d113499889eb4c2aaf270aec" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35907 } } }
I20260808 13:02:35.806536 26144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:35.806344 26145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7385fa92d113499889eb4c2aaf270aec. Latest consensus state: current_term: 1 leader_uuid: "7385fa92d113499889eb4c2aaf270aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7385fa92d113499889eb4c2aaf270aec" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 35907 } } }
I20260808 13:02:35.806655 26145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:35.809787 26153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:35.815281 26153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:35.824477 26153 catalog_manager.cc:1347] Generated new cluster ID: d7f8d374671e4fb2826e78c0a7eb2a7a
I20260808 13:02:35.824590 26153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:35.852118 26153 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:35.854028 26153 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:35.863662 26153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7385fa92d113499889eb4c2aaf270aec: Generated new TSK 0
I20260808 13:02:35.864764 26153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:35.884675 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26038
W20260808 13:02:35.886072 26125 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56331
W20260808 13:02:35.886934 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26038: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 13:02:35.887107 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 26038
W20260808 13:02:35.914167 26012 connection.cc:535] client connection to 127.24.167.62:35907 recv error: Network error: failed to read from TLS socket (remote: 127.24.167.62:35907): Connection reset by peer (error 104)
2026-08-08T13:02:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:36 dist-test-slave-tk9z krb5kdc[26173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:36 dist-test-slave-tk9z krb5kdc[26173](info): set up 2 sockets
Aug 08 13:02:36 dist-test-slave-tk9z krb5kdc[26173](info): commencing operation
krb5kdc: starting...
W20260808 13:02:37.247370 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.314s user 0.003s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:37 dist-test-slave-tk9z krb5kdc[26173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 13:02:37.565826 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:34151
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:34151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:37.925659 26188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:37.926025 26188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:37.926201 26188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:37.933645 26188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:37.933830 26188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:37.933938 26188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:37.934060 26188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:37.934139 26188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:37.944614 26188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:34151
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:34151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:37.946096 26188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:37.947490 26188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:37.959882 26196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:37.959859 26197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:37.960585 26199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:37.961043 26188 server_base.cc:1034] running on GCE node
I20260808 13:02:37.961923 26188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:37.963472 26188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:37.964709 26188 hybrid_clock.cc:648] HybridClock initialized: now 1786194157964673 us; error 35 us; skew 500 ppm
I20260808 13:02:37.965094 26188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:37 dist-test-slave-tk9z krb5kdc[26173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194157, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:37.972023 26188 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:02:37.973635 26188 webserver.cc:460] Webserver started at http://127.24.167.62:38399/ using document root <none> and password file <none>
I20260808 13:02:37.974352 26188 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:37.974568 26188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:37.974905 26188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:37.977995 26188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "8dca51a36bbd4eba9ed53f1046b48803"
format_stamp: "Formatted at 2026-08-08 13:02:37 on dist-test-slave-tk9z"
server_key: "c882797a8fb9218b9632815fc82348bb"
server_key_iv: "020d77caff7eae6fb5287a53b8fe5666"
server_key_version: "encryptionkey@0"
I20260808 13:02:37.978890 26188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "8dca51a36bbd4eba9ed53f1046b48803"
format_stamp: "Formatted at 2026-08-08 13:02:37 on dist-test-slave-tk9z"
server_key: "c882797a8fb9218b9632815fc82348bb"
server_key_iv: "020d77caff7eae6fb5287a53b8fe5666"
server_key_version: "encryptionkey@0"
I20260808 13:02:37.984513 26188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 13:02:37.988200 26209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:37.989696 26188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 13:02:37.989897 26188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "8dca51a36bbd4eba9ed53f1046b48803"
format_stamp: "Formatted at 2026-08-08 13:02:37 on dist-test-slave-tk9z"
server_key: "c882797a8fb9218b9632815fc82348bb"
server_key_iv: "020d77caff7eae6fb5287a53b8fe5666"
server_key_version: "encryptionkey@0"
I20260808 13:02:37.990142 26188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:37.997079 26188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:38.004894 26188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:38.006726 26188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:38.027491 26188 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:34151
I20260808 13:02:38.027529 26280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:34151 every 8 connection(s)
I20260808 13:02:38.029068 26188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:38.029929 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 26188
I20260808 13:02:38.030136 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:38.030715 25244 external_mini_cluster.cc:1428] Setting key e2a85350a5930ba1bc18ab75e2096291
I20260808 13:02:38.033623 26281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:38.040494 26284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:38.032767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40051 (local address 127.24.167.62:34151)
0808 13:02:38.033748 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:38.033765 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:38.034137 (+ 372us) server_negotiation.cc:407] Connection header received
0808 13:02:38.036585 (+ 2448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:38.036612 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:38.037057 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:38.037500 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:38.038529 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:38.038538 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:38.038554 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:38.038571 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:38.038670 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:38.038695 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:38.038909 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:38.039046 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 13:02:38.039235 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":400,"threads_started":1}
I20260808 13:02:38.044399 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:38.048728 26281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803: Bootstrap starting.
I20260808 13:02:38.052460 26284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:38.049822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41751 (local address 127.24.167.62:34151)
0808 13:02:38.050002 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:38.050007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:38.050524 (+ 517us) server_negotiation.cc:407] Connection header received
0808 13:02:38.050645 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:38.050652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:38.050723 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:38.050850 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:38.051783 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:38.051789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:38.051793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:38.051810 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:38.051864 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:38.051883 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:38.051996 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:38.052079 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 13:02:38.052138 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 13:02:38.052573 26281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:38.053960 26281 log.cc:824] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:38.057300 26281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803: No bootstrap required, opened a new log
I20260808 13:02:38.066303 26281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dca51a36bbd4eba9ed53f1046b48803" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34151 } }
I20260808 13:02:38.066766 26281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:38.066857 26281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dca51a36bbd4eba9ed53f1046b48803, State: Initialized, Role: FOLLOWER
I20260808 13:02:38.067416 26281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dca51a36bbd4eba9ed53f1046b48803" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34151 } }
I20260808 13:02:38.067674 26281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:38.067798 26281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:38.067934 26281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:38.070226 26281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dca51a36bbd4eba9ed53f1046b48803" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34151 } }
I20260808 13:02:38.070864 26281 leader_election.cc:302] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dca51a36bbd4eba9ed53f1046b48803; no voters:
I20260808 13:02:38.071448 26281 leader_election.cc:288] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:38.071601 26292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:38.072837 26292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [term 1 LEADER]: Becoming Leader. State: Replica: 8dca51a36bbd4eba9ed53f1046b48803, State: Running, Role: LEADER
I20260808 13:02:38.073448 26292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dca51a36bbd4eba9ed53f1046b48803" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34151 } }
I20260808 13:02:38.073880 26281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:38.076653 26295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dca51a36bbd4eba9ed53f1046b48803. Latest consensus state: current_term: 1 leader_uuid: "8dca51a36bbd4eba9ed53f1046b48803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dca51a36bbd4eba9ed53f1046b48803" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34151 } } }
I20260808 13:02:38.076663 26293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dca51a36bbd4eba9ed53f1046b48803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dca51a36bbd4eba9ed53f1046b48803" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 34151 } } }
I20260808 13:02:38.077184 26293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:38.077781 26295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:38.078016 26302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:38.084209 26302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:38.094341 26302 catalog_manager.cc:1347] Generated new cluster ID: 14adb7cc2ee843a99ac05e2f7a3bfe2e
I20260808 13:02:38.094522 26302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:38.134056 26302 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:38.134629 26302 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:38.141409 26302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dca51a36bbd4eba9ed53f1046b48803: Generated new TSK 0
I20260808 13:02:38.142271 26302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:38.146605 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26188
W20260808 13:02:38.147945 26273 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40051
W20260808 13:02:38.148738 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26188: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 13:02:38.148923 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 26188
2026-08-08T13:02:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2267 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:38 dist-test-slave-tk9z krb5kdc[26318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:38 dist-test-slave-tk9z krb5kdc[26318](info): set up 2 sockets
Aug 08 13:02:38 dist-test-slave-tk9z krb5kdc[26318](info): commencing operation
krb5kdc: starting...
W20260808 13:02:39.489382 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.288s user 0.003s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:39 dist-test-slave-tk9z krb5kdc[26318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 13:02:39.830744 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:44187
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:44187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:40.193749 26333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:40.194070 26333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:40.194175 26333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:40.202502 26333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:40.202633 26333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:40.202685 26333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:40.202754 26333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:40.202800 26333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:40.214162 26333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:44187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:44187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:40.215634 26333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:40.217077 26333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:40.230412 26341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:40.230501 26340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:40.230470 26345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:40.231515 26333 server_base.cc:1034] running on GCE node
I20260808 13:02:40.232501 26333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:40.234220 26333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:40.235528 26333 hybrid_clock.cc:648] HybridClock initialized: now 1786194160235507 us; error 102 us; skew 500 ppm
I20260808 13:02:40.235999 26333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:40 dist-test-slave-tk9z krb5kdc[26318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:40.239756 26333 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:02:40.241415 26333 webserver.cc:460] Webserver started at http://127.24.167.62:36683/ using document root <none> and password file <none>
I20260808 13:02:40.242177 26333 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:40.242295 26333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:40.242661 26333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:40.246178 26333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "cdd9ff8c2df64aee9b48354e229deb64"
format_stamp: "Formatted at 2026-08-08 13:02:40 on dist-test-slave-tk9z"
server_key: "77e074810c688d207f37583b295e3ea7"
server_key_iv: "b56ee8984f141d0aa87d8934552b9ba1"
server_key_version: "encryptionkey@0"
I20260808 13:02:40.247108 26333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "cdd9ff8c2df64aee9b48354e229deb64"
format_stamp: "Formatted at 2026-08-08 13:02:40 on dist-test-slave-tk9z"
server_key: "77e074810c688d207f37583b295e3ea7"
server_key_iv: "b56ee8984f141d0aa87d8934552b9ba1"
server_key_version: "encryptionkey@0"
I20260808 13:02:40.253165 26333 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 13:02:40.257100 26355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:40.258644 26333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:02:40.258854 26333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "cdd9ff8c2df64aee9b48354e229deb64"
format_stamp: "Formatted at 2026-08-08 13:02:40 on dist-test-slave-tk9z"
server_key: "77e074810c688d207f37583b295e3ea7"
server_key_iv: "b56ee8984f141d0aa87d8934552b9ba1"
server_key_version: "encryptionkey@0"
I20260808 13:02:40.259085 26333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:40.300189 26333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:40.304735 26333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:40.305073 26333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:40.326521 26333 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:44187
I20260808 13:02:40.326519 26424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:44187 every 8 connection(s)
I20260808 13:02:40.328066 26333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:40.332114 26425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:40.336176 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 26333
I20260808 13:02:40.336381 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:40.337031 25244 external_mini_cluster.cc:1428] Setting key 5dca5eab2642a70a551d72110374148d
I20260808 13:02:40.347218 26425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64: Bootstrap starting.
I20260808 13:02:40.351413 26425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:40.352869 26425 log.cc:824] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64: Log is configured to *not* fsync() on all Append() calls
Aug 08 13:02:40 dist-test-slave-tk9z krb5kdc[26318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:02:40.356735 26425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64: No bootstrap required, opened a new log
I20260808 13:02:40.362555 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:40.339837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53029 (local address 127.24.167.62:44187)
0808 13:02:40.340708 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:40.340721 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:40.340765 (+ 44us) server_negotiation.cc:407] Connection header received
0808 13:02:40.342680 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:40.342705 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:40.343134 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:40.343499 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:40.345561 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:40.346968 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:40.349025 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:40.349693 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:40.349825 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:40.355276 (+ 5451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:40.355284 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:40.355298 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:40.355331 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:40.358468 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:40.359167 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:40.359179 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:40.359185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:40.359354 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:40.359770 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:40.359777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:40.359788 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:40.360325 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:40.360646 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:40.361110 (+ 464us) server_negotiation.cc:299] Negotiation successful
0808 13:02:40.361526 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":383,"threads_started":1}
I20260808 13:02:40.366636 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:40.366402 26425 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd9ff8c2df64aee9b48354e229deb64" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44187 } }
I20260808 13:02:40.366806 26425 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:40.366931 26425 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd9ff8c2df64aee9b48354e229deb64, State: Initialized, Role: FOLLOWER
I20260808 13:02:40.367592 26425 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd9ff8c2df64aee9b48354e229deb64" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44187 } }
I20260808 13:02:40.367862 26425 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:40.368017 26425 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:40.368216 26425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:40.370882 26425 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd9ff8c2df64aee9b48354e229deb64" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44187 } }
I20260808 13:02:40.371527 26425 leader_election.cc:302] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdd9ff8c2df64aee9b48354e229deb64; no voters:
I20260808 13:02:40.372287 26425 leader_election.cc:288] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:40.372413 26439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:40.373311 26439 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [term 1 LEADER]: Becoming Leader. State: Replica: cdd9ff8c2df64aee9b48354e229deb64, State: Running, Role: LEADER
I20260808 13:02:40.373929 26439 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd9ff8c2df64aee9b48354e229deb64" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44187 } }
I20260808 13:02:40.376622 26425 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:40.376864 26442 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdd9ff8c2df64aee9b48354e229deb64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd9ff8c2df64aee9b48354e229deb64" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44187 } } }
I20260808 13:02:40.377288 26442 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:40.380656 26441 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd9ff8c2df64aee9b48354e229deb64. Latest consensus state: current_term: 1 leader_uuid: "cdd9ff8c2df64aee9b48354e229deb64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd9ff8c2df64aee9b48354e229deb64" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 44187 } } }
I20260808 13:02:40.380937 26441 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:40.386874 26447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:40.391675 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:40.371297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40475 (local address 127.24.167.62:44187)
0808 13:02:40.371519 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:40.371528 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:40.372098 (+ 570us) server_negotiation.cc:407] Connection header received
0808 13:02:40.372331 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:40.372336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:40.372425 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:40.372554 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:40.375094 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:40.376295 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:40.378558 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:40.379517 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:40.379675 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:40.383736 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:40.383746 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:40.383754 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:40.383807 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:40.388019 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:40.389018 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:40.389024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:40.389027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:40.389164 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:40.389638 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:40.389648 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:40.389652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:40.390163 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:40.390454 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:40.391023 (+ 569us) server_negotiation.cc:299] Negotiation successful
0808 13:02:40.391267 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 13:02:40.394877 26447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:40.404989 26447 catalog_manager.cc:1347] Generated new cluster ID: 5720c54f5b334785b4716add93310bc3
I20260808 13:02:40.405134 26447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:40.428202 26447 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:40.429735 26447 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:40.444689 26447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdd9ff8c2df64aee9b48354e229deb64: Generated new TSK 0
I20260808 13:02:40.445891 26447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:40.458887 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26333
I20260808 13:02:40.460014 26420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:40.564873 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 26333
2026-08-08T13:02:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:40 dist-test-slave-tk9z krb5kdc[26475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:40 dist-test-slave-tk9z krb5kdc[26475](info): set up 2 sockets
Aug 08 13:02:40 dist-test-slave-tk9z krb5kdc[26475](info): commencing operation
krb5kdc: starting...
W20260808 13:02:41.928541 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.315s user 0.006s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:42 dist-test-slave-tk9z krb5kdc[26475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 13:02:42.269783 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:37629
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:37629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:42.626243 26493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:42.626704 26493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:42.626884 26493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:42.634356 26493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:42.634572 26493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:42.634682 26493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:42.634799 26493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:42.634892 26493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:42.645295 26493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:37629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:37629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:42.646799 26493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:42.648216 26493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:42.660935 26498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:42.660952 26503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:42.660981 26500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:42.661849 26493 server_base.cc:1034] running on GCE node
I20260808 13:02:42.662748 26493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:42.664234 26493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:42.665460 26493 hybrid_clock.cc:648] HybridClock initialized: now 1786194162665415 us; error 41 us; skew 500 ppm
I20260808 13:02:42.665863 26493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:42 dist-test-slave-tk9z krb5kdc[26475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194162, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:42.669275 26493 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:02:42.670887 26493 webserver.cc:460] Webserver started at http://127.24.167.62:45619/ using document root <none> and password file <none>
I20260808 13:02:42.671594 26493 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:42.671710 26493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:42.671984 26493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:42.675185 26493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "4dbd67b74d4449a4af7cc6963f421e5e"
format_stamp: "Formatted at 2026-08-08 13:02:42 on dist-test-slave-tk9z"
server_key: "5567e03f866353b313102414e37d18b0"
server_key_iv: "eda37a191f10bdeb6807488dfee2e27e"
server_key_version: "encryptionkey@0"
I20260808 13:02:42.675964 26493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "4dbd67b74d4449a4af7cc6963f421e5e"
format_stamp: "Formatted at 2026-08-08 13:02:42 on dist-test-slave-tk9z"
server_key: "5567e03f866353b313102414e37d18b0"
server_key_iv: "eda37a191f10bdeb6807488dfee2e27e"
server_key_version: "encryptionkey@0"
I20260808 13:02:42.681663 26493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 13:02:42.685653 26511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:42.687186 26493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 13:02:42.687397 26493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "4dbd67b74d4449a4af7cc6963f421e5e"
format_stamp: "Formatted at 2026-08-08 13:02:42 on dist-test-slave-tk9z"
server_key: "5567e03f866353b313102414e37d18b0"
server_key_iv: "eda37a191f10bdeb6807488dfee2e27e"
server_key_version: "encryptionkey@0"
I20260808 13:02:42.687633 26493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:42.713930 26493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:42.718400 26493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:42.718721 26493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:42.738458 26493 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:37629
I20260808 13:02:42.738481 26581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:37629 every 8 connection(s)
I20260808 13:02:42.739916 26493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:42.743194 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 26493
I20260808 13:02:42.743454 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:42.743974 25244 external_mini_cluster.cc:1428] Setting key 7f4dca15ac497999393a0e3ec957329a
I20260808 13:02:42.744254 26582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:42.759951 26582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e: Bootstrap starting.
I20260808 13:02:42.760102 26587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:42.745929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55355 (local address 127.24.167.62:37629)
0808 13:02:42.746822 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:42.746837 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:42.746891 (+ 54us) server_negotiation.cc:407] Connection header received
0808 13:02:42.749312 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:42.749344 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:42.749908 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:42.750435 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:42.751013 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:42.753097 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:42.754650 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:42.755756 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:42.755954 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:42.757218 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:42.757229 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:42.757248 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:42.757994 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:42.758096 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:42.758126 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:42.758239 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:42.758498 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 13:02:42.758772 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":401,"threads_started":1}
I20260808 13:02:42.763360 26582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:42.764073 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:42.764578 26582 log.cc:824] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:42.767385 26582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e: No bootstrap required, opened a new log
I20260808 13:02:42.776235 26587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:42.769022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48013 (local address 127.24.167.62:37629)
0808 13:02:42.769298 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:42.769307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:42.769669 (+ 362us) server_negotiation.cc:407] Connection header received
0808 13:02:42.769866 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:42.769870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:42.769938 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:42.770033 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:42.770715 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:42.771525 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:42.773430 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:42.773969 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:42.774085 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:42.775369 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:42.775374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:42.775378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:42.775398 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:42.775470 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:42.775488 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:42.775567 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:42.775746 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 13:02:42.775829 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 13:02:42.777650 26582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbd67b74d4449a4af7cc6963f421e5e" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 37629 } }
I20260808 13:02:42.778195 26582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:42.778316 26582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbd67b74d4449a4af7cc6963f421e5e, State: Initialized, Role: FOLLOWER
I20260808 13:02:42.779022 26582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbd67b74d4449a4af7cc6963f421e5e" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 37629 } }
I20260808 13:02:42.779273 26582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:42.779410 26582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:42.779588 26582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:42.781800 26582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbd67b74d4449a4af7cc6963f421e5e" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 37629 } }
I20260808 13:02:42.782368 26582 leader_election.cc:302] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dbd67b74d4449a4af7cc6963f421e5e; no voters:
I20260808 13:02:42.783037 26582 leader_election.cc:288] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:42.783213 26597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:42.784654 26597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [term 1 LEADER]: Becoming Leader. State: Replica: 4dbd67b74d4449a4af7cc6963f421e5e, State: Running, Role: LEADER
I20260808 13:02:42.785256 26597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbd67b74d4449a4af7cc6963f421e5e" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 37629 } }
I20260808 13:02:42.785571 26582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:42.788534 26598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dbd67b74d4449a4af7cc6963f421e5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbd67b74d4449a4af7cc6963f421e5e" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 37629 } } }
I20260808 13:02:42.789000 26598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:42.789613 26600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dbd67b74d4449a4af7cc6963f421e5e. Latest consensus state: current_term: 1 leader_uuid: "4dbd67b74d4449a4af7cc6963f421e5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbd67b74d4449a4af7cc6963f421e5e" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 37629 } } }
I20260808 13:02:42.789988 26600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:42.789985 26606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:42.799882 26606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:42.812788 26606 catalog_manager.cc:1347] Generated new cluster ID: 9efa20fce377462d83ae3664f0bcdac8
I20260808 13:02:42.812948 26606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:42.838320 26606 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:42.840047 26606 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:42.846290 26606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dbd67b74d4449a4af7cc6963f421e5e: Generated new TSK 0
I20260808 13:02:42.847219 26606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:42.859262 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26493
W20260808 13:02:42.860455 26575 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55355
W20260808 13:02:42.861228 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26493: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 13:02:42.861399 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 26493
2026-08-08T13:02:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:42 dist-test-slave-tk9z krb5kdc[26624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:42 dist-test-slave-tk9z krb5kdc[26624](info): set up 2 sockets
Aug 08 13:02:42 dist-test-slave-tk9z krb5kdc[26624](info): commencing operation
krb5kdc: starting...
W20260808 13:02:44.220268 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.313s user 0.002s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:44 dist-test-slave-tk9z krb5kdc[26624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 13:02:44.559791 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:46365
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:46365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:44.936569 26641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:44.936898 26641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:44.937090 26641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:44.944696 26641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:44.944828 26641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:44.944963 26641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:44.945076 26641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:44.945156 26641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:44.955649 26641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:46365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:46365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:44.957104 26641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:44.958575 26641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:44.971573 26649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:44.971642 26652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:44.971637 26650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:44.973184 26641 server_base.cc:1034] running on GCE node
I20260808 13:02:44.974130 26641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:44.975792 26641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:44.977042 26641 hybrid_clock.cc:648] HybridClock initialized: now 1786194164976989 us; error 48 us; skew 500 ppm
I20260808 13:02:44.977447 26641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:44 dist-test-slave-tk9z krb5kdc[26624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194164, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:44.985440 26641 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:02:44.987339 26641 webserver.cc:460] Webserver started at http://127.24.167.62:43459/ using document root <none> and password file <none>
I20260808 13:02:44.988062 26641 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:44.988176 26641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:44.988448 26641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:44.991694 26641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "0bb2e8a1f85d4e7083da205505565c9b"
format_stamp: "Formatted at 2026-08-08 13:02:44 on dist-test-slave-tk9z"
server_key: "4f22a5fc7c003f78026832a25b1f23a4"
server_key_iv: "047b59f00cf67020878ac905447dfac7"
server_key_version: "encryptionkey@0"
I20260808 13:02:44.992445 26641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "0bb2e8a1f85d4e7083da205505565c9b"
format_stamp: "Formatted at 2026-08-08 13:02:44 on dist-test-slave-tk9z"
server_key: "4f22a5fc7c003f78026832a25b1f23a4"
server_key_iv: "047b59f00cf67020878ac905447dfac7"
server_key_version: "encryptionkey@0"
I20260808 13:02:44.998049 26641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 13:02:45.001878 26662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:45.003518 26641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 13:02:45.003749 26641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "0bb2e8a1f85d4e7083da205505565c9b"
format_stamp: "Formatted at 2026-08-08 13:02:44 on dist-test-slave-tk9z"
server_key: "4f22a5fc7c003f78026832a25b1f23a4"
server_key_iv: "047b59f00cf67020878ac905447dfac7"
server_key_version: "encryptionkey@0"
I20260808 13:02:45.004021 26641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:45.010973 26641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:45.015730 26641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:45.016109 26641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:45.037842 26641 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:46365
I20260808 13:02:45.037879 26722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:46365 every 8 connection(s)
I20260808 13:02:45.039489 26641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:45.043906 26723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:45.046180 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 26641
I20260808 13:02:45.046440 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:45.047048 25244 external_mini_cluster.cc:1428] Setting key 65088fd6562a1552284218887135098e
I20260808 13:02:45.055941 26729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:45.049077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35899 (local address 127.24.167.62:46365)
0808 13:02:45.050105 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:45.050121 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:45.050175 (+ 54us) server_negotiation.cc:407] Connection header received
0808 13:02:45.052073 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:45.052099 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:45.052512 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:45.052894 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:45.054163 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:45.054170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:45.054183 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:45.054201 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:45.054271 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:45.054293 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:45.054596 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:45.054709 (+ 113us) server_negotiation.cc:299] Negotiation successful
0808 13:02:45.054950 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":506,"threads_started":1}
I20260808 13:02:45.059768 26723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b: Bootstrap starting.
I20260808 13:02:45.059948 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:45.064262 26723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:45.065872 26723 log.cc:824] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:45.068827 26729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:45.065884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33873 (local address 127.24.167.62:46365)
0808 13:02:45.066064 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:45.066072 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:45.066516 (+ 444us) server_negotiation.cc:407] Connection header received
0808 13:02:45.066782 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:45.066789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:45.066858 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:45.067012 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:45.068074 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:45.068079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:45.068083 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 13:02:45.068101 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:45.068150 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 13:02:45.068170 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:45.068297 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:45.068472 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 13:02:45.068546 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 13:02:45.069542 26723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b: No bootstrap required, opened a new log
I20260808 13:02:45.078006 26723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb2e8a1f85d4e7083da205505565c9b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46365 } }
I20260808 13:02:45.078433 26723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 13:02:45.078537 26723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb2e8a1f85d4e7083da205505565c9b, State: Initialized, Role: FOLLOWER
I20260808 13:02:45.079202 26723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb2e8a1f85d4e7083da205505565c9b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46365 } }
I20260808 13:02:45.079428 26723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 13:02:45.079535 26723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 13:02:45.079663 26723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:45.082494 26723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb2e8a1f85d4e7083da205505565c9b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46365 } }
I20260808 13:02:45.083026 26723 leader_election.cc:302] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bb2e8a1f85d4e7083da205505565c9b; no voters:
I20260808 13:02:45.083637 26723 leader_election.cc:288] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 13:02:45.083811 26744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:45.085937 26723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:45.086771 26744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [term 1 LEADER]: Becoming Leader. State: Replica: 0bb2e8a1f85d4e7083da205505565c9b, State: Running, Role: LEADER
I20260808 13:02:45.087520 26744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb2e8a1f85d4e7083da205505565c9b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46365 } }
I20260808 13:02:45.090893 26746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bb2e8a1f85d4e7083da205505565c9b. Latest consensus state: current_term: 1 leader_uuid: "0bb2e8a1f85d4e7083da205505565c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb2e8a1f85d4e7083da205505565c9b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46365 } } }
I20260808 13:02:45.090909 26745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bb2e8a1f85d4e7083da205505565c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb2e8a1f85d4e7083da205505565c9b" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46365 } } }
I20260808 13:02:45.091471 26745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:45.091473 26746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:45.093007 26755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:45.098615 26755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:45.108610 26755 catalog_manager.cc:1347] Generated new cluster ID: 4bcaa1e8e337410fa8304f9ed24c523f
I20260808 13:02:45.108755 26755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:45.121701 26755 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:45.122247 26755 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:45.133481 26755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bb2e8a1f85d4e7083da205505565c9b: Generated new TSK 0
I20260808 13:02:45.134637 26755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:45.144034 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26641
W20260808 13:02:45.145807 26717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35899
W20260808 13:02:45.146678 25244 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26641: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 13:02:45.146869 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 26641
2026-08-08T13:02:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2288 ms)
[----------] 18 tests from AuthTokenIssuingTest (41539 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:45 dist-test-slave-tk9z krb5kdc[26771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:45 dist-test-slave-tk9z krb5kdc[26771](info): set up 2 sockets
Aug 08 13:02:45 dist-test-slave-tk9z krb5kdc[26771](info): commencing operation
krb5kdc: starting...
W20260808 13:02:46.500195 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.304s user 0.001s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:46 dist-test-slave-tk9z krb5kdc[26771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 13:02:46.845489 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:46549
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:46549,127.24.167.61:36121,127.24.167.60:38777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:47.206121 26789 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 13:02:47.206535 26789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:47.206622 26789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:47.206714 26789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:47.214051 26789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:47.214169 26789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:47.214231 26789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:47.214282 26789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:47.214347 26789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:47.224682 26789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:46549,127.24.167.61:36121,127.24.167.60:38777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:46549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:47.226058 26789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:47.227455 26789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:47.240346 26795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:47.240347 26799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:47.240381 26796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:47.241328 26789 server_base.cc:1034] running on GCE node
I20260808 13:02:47.242233 26789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:47.243844 26789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:47.245106 26789 hybrid_clock.cc:648] HybridClock initialized: now 1786194167245027 us; error 61 us; skew 500 ppm
I20260808 13:02:47.245464 26789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:47 dist-test-slave-tk9z krb5kdc[26771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194167, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:47.249004 26789 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:02:47.250698 26789 webserver.cc:460] Webserver started at http://127.24.167.62:35673/ using document root <none> and password file <none>
I20260808 13:02:47.251462 26789 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:47.251559 26789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:47.251948 26789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:47.255199 26789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "a4d3aaf594294ae08c9dafd0cb46e917"
format_stamp: "Formatted at 2026-08-08 13:02:47 on dist-test-slave-tk9z"
server_key: "432fa722e2721d0695c21ee9182a0564"
server_key_iv: "511f1e6a5b31e9a2e14bd05b94cc4afc"
server_key_version: "encryptionkey@0"
I20260808 13:02:47.256037 26789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "a4d3aaf594294ae08c9dafd0cb46e917"
format_stamp: "Formatted at 2026-08-08 13:02:47 on dist-test-slave-tk9z"
server_key: "432fa722e2721d0695c21ee9182a0564"
server_key_iv: "511f1e6a5b31e9a2e14bd05b94cc4afc"
server_key_version: "encryptionkey@0"
I20260808 13:02:47.261358 26789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 13:02:47.265010 26809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:47.266486 26789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 13:02:47.266683 26789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "a4d3aaf594294ae08c9dafd0cb46e917"
format_stamp: "Formatted at 2026-08-08 13:02:47 on dist-test-slave-tk9z"
server_key: "432fa722e2721d0695c21ee9182a0564"
server_key_iv: "511f1e6a5b31e9a2e14bd05b94cc4afc"
server_key_version: "encryptionkey@0"
I20260808 13:02:47.266916 26789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:47.286267 26789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:47.290845 26789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:47.291159 26789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:47.312091 26789 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:46549
I20260808 13:02:47.312098 26868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:46549 every 8 connection(s)
I20260808 13:02:47.313825 26789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:47.318189 26869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:47.320257 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 26789
I20260808 13:02:47.320533 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:47.321272 25244 external_mini_cluster.cc:1428] Setting key 69058d08c858372cbfe834c332002f4e
I20260808 13:02:47.325413 26869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 13:02:47 dist-test-slave-tk9z krb5kdc[26771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194167, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:02:47.353330 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.328827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:46549 (local address 127.0.0.1:45168)
0808 13:02:47.329765 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.329834 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:47.333099 (+ 3265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:47.334098 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:47.334108 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:47.335099 (+ 991us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:47.335698 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:47.335729 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:47.338178 (+ 2449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:47.338182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:47.339490 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:47.339504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:47.340903 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:47.340909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:47.341075 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:47.341811 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:47.341835 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:47.344250 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:47.348935 (+ 4685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:47.348948 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:47.348958 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:47.349371 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:47.349914 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:47.349917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:47.349919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:47.350085 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:47.351216 (+ 1131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:47.351222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:47.351408 (+ 186us) client_negotiation.cc:769] Sending connection context
0808 13:02:47.351652 (+ 244us) client_negotiation.cc:240] Negotiation successful
0808 13:02:47.352099 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":341,"threads_started":1}
I20260808 13:02:47.353434 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.329926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.24.167.62:46549)
0808 13:02:47.330589 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:47.330595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:47.330637 (+ 42us) server_negotiation.cc:407] Connection header received
0808 13:02:47.333570 (+ 2933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:47.333575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:47.333827 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:47.333998 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:47.336003 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:47.337996 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:47.339648 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:47.340666 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:47.340880 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:47.344538 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:47.344552 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:47.344572 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:47.344617 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:47.348785 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:47.349528 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:47.349540 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:47.349546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:47.349777 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:47.350212 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:47.350220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:47.350226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:47.351052 (+ 826us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:47.351273 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:47.351862 (+ 589us) server_negotiation.cc:299] Negotiation successful
0808 13:02:47.352316 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":504,"thread_start_us":339,"threads_started":1}
I20260808 13:02:47.356921 26869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.167.61@KRBTEST.COM; defaulting to no policy
I20260808 13:02:47.358134 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.357684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:36121 (local address 127.0.0.1:33400)
0808 13:02:47.357852 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.358019 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 13:02:47.358501 26814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.61@KRBTEST.COM" created.
I20260808 13:02:47.360164 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.359795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:36121 (local address 127.0.0.1:33410)
0808 13:02:47.359977 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.360051 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 13:02:47.360445 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.61:36121: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.61.keytab.
Entry for principal kudu/127.24.167.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.61.keytab.
I20260808 13:02:47.397112 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.61:36121
--webserver_interface=127.24.167.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.61.keytab
--principal=kudu/127.24.167.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:46549,127.24.167.61:36121,127.24.167.60:38777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 13:02:47.409814 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } attempt: 1
I20260808 13:02:47.411180 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.410712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:36121 (local address 127.0.0.1:33412)
0808 13:02:47.410918 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.411035 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260808 13:02:47.412231 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.411865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:36121 (local address 127.0.0.1:33426)
0808 13:02:47.412016 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.412097 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 13:02:47.412453 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.61:36121: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
I20260808 13:02:47.480695 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } attempt: 2
I20260808 13:02:47.482036 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.481494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:36121 (local address 127.0.0.1:33428)
0808 13:02:47.481755 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.481861 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 13:02:47.483232 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.482822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:36121 (local address 127.0.0.1:33444)
0808 13:02:47.482993 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.483093 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260808 13:02:47.483413 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.61:36121: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
I20260808 13:02:47.574661 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } attempt: 3
I20260808 13:02:47.575968 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.575453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:36121 (local address 127.0.0.1:33460)
0808 13:02:47.575706 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.575824 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 13:02:47.577049 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.576649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:36121 (local address 127.0.0.1:33472)
0808 13:02:47.576816 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.576886 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 13:02:47.577235 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.61:36121: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
I20260808 13:02:47.720531 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } attempt: 4
I20260808 13:02:47.721956 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.721335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:36121 (local address 127.0.0.1:33486)
0808 13:02:47.721701 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.721797 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
I20260808 13:02:47.723122 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.722736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:36121 (local address 127.0.0.1:33496)
0808 13:02:47.722886 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.722997 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 13:02:47.723367 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.61:36121: Network error: Client connection negotiation failed: client connection to 127.24.167.61:36121: connect: Connection refused (error 111)
W20260808 13:02:47.756191 26878 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 13:02:47.756644 26878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:47.756781 26878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:47.756924 26878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:47.764727 26878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:47.764856 26878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:47.764986 26878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:47.765090 26878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:47.765196 26878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:47.775797 26878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:46549,127.24.167.61:36121,127.24.167.60:38777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.61.keytab
--principal=kudu/127.24.167.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.61:36121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.167.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:47.777208 26878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:47.778704 26878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:47.791041 26885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:47.791281 26889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:47.791325 26886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:47.792246 26878 server_base.cc:1034] running on GCE node
I20260808 13:02:47.793176 26878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:47.795010 26878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:47.796247 26878 hybrid_clock.cc:648] HybridClock initialized: now 1786194167796147 us; error 67 us; skew 500 ppm
I20260808 13:02:47.796834 26878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:47 dist-test-slave-tk9z krb5kdc[26771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194167, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:47.800403 26878 init.cc:377] Logged in from keytab as kudu/127.24.167.61@KRBTEST.COM (short username kudu)
I20260808 13:02:47.801888 26878 webserver.cc:460] Webserver started at http://127.24.167.61:38983/ using document root <none> and password file <none>
I20260808 13:02:47.802637 26878 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:47.802731 26878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:47.803017 26878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:47.805994 26878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/data/instance:
uuid: "f4d647d20e5a4caea73e963ab877c1f6"
format_stamp: "Formatted at 2026-08-08 13:02:47 on dist-test-slave-tk9z"
server_key: "47d7c1764d418d59608efc4e25e749e9"
server_key_iv: "a2cb3f63659a8e321e71d6c6b0e74189"
server_key_version: "encryptionkey@0"
I20260808 13:02:47.806840 26878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/wal/instance:
uuid: "f4d647d20e5a4caea73e963ab877c1f6"
format_stamp: "Formatted at 2026-08-08 13:02:47 on dist-test-slave-tk9z"
server_key: "47d7c1764d418d59608efc4e25e749e9"
server_key_iv: "a2cb3f63659a8e321e71d6c6b0e74189"
server_key_version: "encryptionkey@0"
I20260808 13:02:47.812337 26878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 13:02:47.816102 26899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:47.817608 26878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 13:02:47.817827 26878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/wal
uuid: "f4d647d20e5a4caea73e963ab877c1f6"
format_stamp: "Formatted at 2026-08-08 13:02:47 on dist-test-slave-tk9z"
server_key: "47d7c1764d418d59608efc4e25e749e9"
server_key_iv: "a2cb3f63659a8e321e71d6c6b0e74189"
server_key_version: "encryptionkey@0"
I20260808 13:02:47.818090 26878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:47.846685 26878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:47.851558 26878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:47.851861 26878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:47.872180 26878 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.61:36121
I20260808 13:02:47.872205 26969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.61:36121 every 8 connection(s)
I20260808 13:02:47.873696 26878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/data/info.pb
I20260808 13:02:47.877729 26970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:47.881511 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 26878
I20260808 13:02:47.881708 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-1/wal/instance
I20260808 13:02:47.882154 25244 external_mini_cluster.cc:1428] Setting key 6dfdeb5c676ba7734aa4d6640fcd63c3
I20260808 13:02:47.885077 26970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 13:02:47 dist-test-slave-tk9z krb5kdc[26771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194167, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.61@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:02:47.909374 26972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.888089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45176 (local address 127.24.167.62:46549)
0808 13:02:47.888714 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:47.888724 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:47.889701 (+ 977us) server_negotiation.cc:407] Connection header received
0808 13:02:47.891944 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:47.891952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:47.892064 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:47.892206 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:47.893939 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:47.895254 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:47.897182 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:47.897777 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:47.897888 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:47.902740 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:47.902745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:47.902749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:47.902790 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:47.905696 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:47.906766 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:47.906773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:47.906776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:47.906929 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:47.907421 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:47.907427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:47.907430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:47.907756 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:47.907938 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:47.908718 (+ 780us) server_negotiation.cc:299] Negotiation successful
0808 13:02:47.908978 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":300,"threads_started":1}
I20260808 13:02:47.910524 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.888678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:46549 (local address 127.0.0.1:45176)
0808 13:02:47.889507 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.889563 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:47.891630 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:47.892469 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:47.892477 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:47.893042 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:47.893675 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:47.893695 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:47.895499 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:47.895507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:47.897002 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:47.897016 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:47.898072 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:47.898083 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:47.898517 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:47.899323 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:47.899359 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:47.902466 (+ 3107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:47.905915 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:47.905926 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:47.905953 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:47.906543 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:47.907058 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:47.907064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:47.907073 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:47.907268 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:47.907909 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:47.907918 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:47.908326 (+ 408us) client_negotiation.cc:769] Sending connection context
0808 13:02:47.908614 (+ 288us) client_negotiation.cc:240] Negotiation successful
0808 13:02:47.909194 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":355,"threads_started":1}
I20260808 13:02:47.912366 26970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.167.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.60@KRBTEST.COM" created.
Aug 08 13:02:47 dist-test-slave-tk9z krb5kdc[26771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194167, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.61@KRBTEST.COM for kudu/127.24.167.61@KRBTEST.COM
I20260808 13:02:47.930655 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.913286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:36121 (local address 127.0.0.1:33504)
0808 13:02:47.913528 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.913556 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:47.913700 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:47.914682 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:47.914686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:47.915075 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:47.915222 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:47.915231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:47.917136 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:47.917144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:47.918117 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:47.918137 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:47.919522 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:47.919531 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:47.919766 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:47.920526 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:47.920558 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:47.923711 (+ 3153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:47.927788 (+ 4077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:47.927796 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:47.927801 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:47.928209 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:47.928657 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:47.928662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:47.928665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:47.928811 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:47.929521 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:47.929526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:47.929680 (+ 154us) client_negotiation.cc:769] Sending connection context
0808 13:02:47.929860 (+ 180us) client_negotiation.cc:240] Negotiation successful
0808 13:02:47.930181 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 13:02:47.930783 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.913312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.24.167.61:36121)
0808 13:02:47.914067 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:47.914077 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:47.914112 (+ 35us) server_negotiation.cc:407] Connection header received
0808 13:02:47.914172 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:47.914181 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:47.914490 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:47.914653 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:47.915418 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:47.916949 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:47.918326 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:47.919270 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:47.919396 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:47.923969 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:47.923984 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:47.924001 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:47.924034 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:47.927609 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:47.928387 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:47.928399 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:47.928401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:47.928527 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:47.928929 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:47.928936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:47.928939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:47.929355 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:47.929600 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:47.930017 (+ 417us) server_negotiation.cc:299] Negotiation successful
0808 13:02:47.930341 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":538,"thread_start_us":334,"threads_started":1}
I20260808 13:02:47.933661 26970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } has no permanent_uuid. Determining permanent_uuid...
I20260808 13:02:47.934968 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.934478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47748)
0808 13:02:47.934651 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.934798 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260808 13:02:47.935243 26902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
I20260808 13:02:47.936830 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.936484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47764)
0808 13:02:47.936646 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.936727 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 13:02:47.937091 26970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:38777: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.60.keytab.
Entry for principal kudu/127.24.167.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.60.keytab.
I20260808 13:02:47.961200 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.60:38777
--webserver_interface=127.24.167.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.60.keytab
--principal=kudu/127.24.167.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:46549,127.24.167.61:36121,127.24.167.60:38777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 13:02:47.986472 26970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } attempt: 1
I20260808 13:02:47.987802 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.987319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47768)
0808 13:02:47.987544 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.987651 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 13:02:47.988852 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:47.988513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47770)
0808 13:02:47.988658 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:47.988748 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 13:02:47.989102 26970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:38777: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
I20260808 13:02:48.022766 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } attempt: 5
Aug 08 13:02:48 dist-test-slave-tk9z krb5kdc[26771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194167, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for kudu/127.24.167.61@KRBTEST.COM
I20260808 13:02:48.043673 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.024098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.24.167.61:36121)
0808 13:02:48.024321 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.024329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.024356 (+ 27us) server_negotiation.cc:407] Connection header received
0808 13:02:48.024441 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.024449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.024605 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.024745 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.026295 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.027655 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.033089 (+ 5434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.033541 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.033708 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.037813 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:48.037820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:48.037825 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:48.037876 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:48.040859 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.041584 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.041590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.041594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.041735 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.042158 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.042163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.042166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.042539 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:48.042694 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:48.043113 (+ 419us) server_negotiation.cc:299] Negotiation successful
0808 13:02:48.043342 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 13:02:48.043738 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.023696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:36121 (local address 127.0.0.1:33510)
0808 13:02:48.023966 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.023993 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:48.024157 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:48.024850 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:48.024858 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:48.025800 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:48.026119 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:48.026135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.027813 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.027818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:48.029307 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:48.029324 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.033736 (+ 4412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.033742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:48.034015 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.034838 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:48.034867 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:48.037542 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:48.041029 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:48.041035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:48.041040 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:48.041438 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:48.041868 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:48.041874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:48.041878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:48.042040 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:48.042673 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:48.042678 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:48.042848 (+ 170us) client_negotiation.cc:769] Sending connection context
0808 13:02:48.043004 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 13:02:48.043336 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 13:02:48.045090 26869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } has no permanent_uuid. Determining permanent_uuid...
I20260808 13:02:48.046157 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.045777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47786)
0808 13:02:48.045940 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.046018 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 13:02:48.047115 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.046784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47798)
0808 13:02:48.046924 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.047018 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 13:02:48.047348 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:38777: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
I20260808 13:02:48.057325 26970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } attempt: 2
I20260808 13:02:48.058640 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.058118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47802)
0808 13:02:48.058308 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.058483 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 13:02:48.059677 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.059325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47816)
0808 13:02:48.059504 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.059570 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 13:02:48.059872 26970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:38777: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
I20260808 13:02:48.098568 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } attempt: 1
I20260808 13:02:48.099895 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.099260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47832)
0808 13:02:48.099509 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.099676 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 13:02:48.101027 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.100642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47840)
0808 13:02:48.100796 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.100880 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 13:02:48.101269 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:38777: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
I20260808 13:02:48.151083 26970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } attempt: 3
I20260808 13:02:48.152377 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.151791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47856)
0808 13:02:48.152058 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.152174 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 13:02:48.153566 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.153087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47858)
0808 13:02:48.153335 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.153434 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260808 13:02:48.153800 26970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:38777: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
I20260808 13:02:48.169519 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } attempt: 2
I20260808 13:02:48.170931 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.170244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47862)
0808 13:02:48.170541 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.170673 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260808 13:02:48.172231 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.171728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47876)
0808 13:02:48.171958 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.172067 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 13:02:48.172478 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:38777: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
I20260808 13:02:48.278720 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } attempt: 3
I20260808 13:02:48.279986 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.279403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47888)
0808 13:02:48.279627 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.279748 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 13:02:48.281157 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.280667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47900)
0808 13:02:48.280883 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.281011 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260808 13:02:48.281402 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:38777: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
I20260808 13:02:48.297034 26970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } attempt: 4
I20260808 13:02:48.298341 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.297752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47912)
0808 13:02:48.298021 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.298134 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 13:02:48.299544 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.299131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47920)
0808 13:02:48.299307 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.299404 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 13:02:48.299822 26970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:38777: Network error: Client connection negotiation failed: client connection to 127.24.167.60:38777: connect: Connection refused (error 111)
W20260808 13:02:48.327968 26979 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 13:02:48.328366 26979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:48.328505 26979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:48.328642 26979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:48.336773 26979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:48.336903 26979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:48.337028 26979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:48.337116 26979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:48.337224 26979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:48.347465 26979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:46549,127.24.167.61:36121,127.24.167.60:38777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.60.keytab
--principal=kudu/127.24.167.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.60:38777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.167.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:48.348862 26979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:48.350193 26979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:48.361918 26985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:48.361980 26986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:48.362031 26988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:48.363261 26979 server_base.cc:1034] running on GCE node
I20260808 13:02:48.364084 26979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:48.365578 26979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:48.366823 26979 hybrid_clock.cc:648] HybridClock initialized: now 1786194168366756 us; error 47 us; skew 500 ppm
I20260808 13:02:48.367166 26979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:48 dist-test-slave-tk9z krb5kdc[26771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194168, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:48.370515 26979 init.cc:377] Logged in from keytab as kudu/127.24.167.60@KRBTEST.COM (short username kudu)
I20260808 13:02:48.372143 26979 webserver.cc:460] Webserver started at http://127.24.167.60:38961/ using document root <none> and password file <none>
I20260808 13:02:48.372851 26979 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:48.372997 26979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:48.373338 26979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:48.376413 26979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/data/instance:
uuid: "77907cb6a96e4926a062b67d418a464a"
format_stamp: "Formatted at 2026-08-08 13:02:48 on dist-test-slave-tk9z"
server_key: "03d5427ddbef6f070635f3ba2f2ffcc2"
server_key_iv: "a11b198958eb5767380cb572ec4ac798"
server_key_version: "encryptionkey@0"
I20260808 13:02:48.377274 26979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/wal/instance:
uuid: "77907cb6a96e4926a062b67d418a464a"
format_stamp: "Formatted at 2026-08-08 13:02:48 on dist-test-slave-tk9z"
server_key: "03d5427ddbef6f070635f3ba2f2ffcc2"
server_key_iv: "a11b198958eb5767380cb572ec4ac798"
server_key_version: "encryptionkey@0"
I20260808 13:02:48.382817 26979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 13:02:48.386471 26998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:48.387823 26979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 13:02:48.388031 26979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/wal
uuid: "77907cb6a96e4926a062b67d418a464a"
format_stamp: "Formatted at 2026-08-08 13:02:48 on dist-test-slave-tk9z"
server_key: "03d5427ddbef6f070635f3ba2f2ffcc2"
server_key_iv: "a11b198958eb5767380cb572ec4ac798"
server_key_version: "encryptionkey@0"
I20260808 13:02:48.388268 26979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:48.405572 26979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:48.409752 26979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:48.410066 26979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:48.430194 26979 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.60:38777
I20260808 13:02:48.430215 27074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.60:38777 every 8 connection(s)
I20260808 13:02:48.431710 26979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/data/info.pb
I20260808 13:02:48.435708 27075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:48.436028 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 26979
I20260808 13:02:48.436247 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786193784050718-25244-0/minicluster-data/master-2/wal/instance
I20260808 13:02:48.436719 25244 external_mini_cluster.cc:1428] Setting key 29ff6857f1c5452d2c1fd9900505d6e8
I20260808 13:02:48.446276 27075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 13:02:48 dist-test-slave-tk9z krb5kdc[26771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.60@KRBTEST.COM
Aug 08 13:02:48 dist-test-slave-tk9z krb5kdc[26771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194168, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.60@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:02:48.458636 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } attempt: 4
I20260808 13:02:48.462670 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.438772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47930 (local address 127.24.167.60:38777)
0808 13:02:48.439642 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.439658 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.439714 (+ 56us) server_negotiation.cc:407] Connection header received
0808 13:02:48.441371 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.441406 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.441951 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.442441 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.443823 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.445730 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.447114 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.448110 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.448316 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.453349 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:48.453357 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:48.453368 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:48.453411 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:48.456889 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.457818 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.457829 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.457840 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.458088 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.458574 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.458581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.458591 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.459343 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:48.459704 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:48.460735 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0808 13:02:48.461089 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":356,"threads_started":1}
Aug 08 13:02:48 dist-test-slave-tk9z krb5kdc[26771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194167, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for kudu/127.24.167.60@KRBTEST.COM
I20260808 13:02:48.468858 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.449259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:46549 (local address 127.0.0.1:45192)
0808 13:02:48.450165 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.450201 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:48.450443 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:48.450912 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:48.450920 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:48.451736 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:48.452158 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:48.452177 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.453261 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.453265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:48.454635 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:48.454651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.455684 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.455689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:48.455820 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.456425 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:48.456446 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:48.459302 (+ 2856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:48.463988 (+ 4686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:48.464001 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:48.464008 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:48.464580 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:48.464975 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:48.464981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:48.464994 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:48.465197 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:48.467626 (+ 2429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:48.467644 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:48.467866 (+ 222us) client_negotiation.cc:769] Sending connection context
0808 13:02:48.468084 (+ 218us) client_negotiation.cc:240] Negotiation successful
0808 13:02:48.468445 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":542,"threads_started":1}
I20260808 13:02:48.468966 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.449239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.24.167.62:46549)
0808 13:02:48.449832 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.449840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.450322 (+ 482us) server_negotiation.cc:407] Connection header received
0808 13:02:48.450589 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.450596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.450685 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.450812 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.452336 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.453075 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.454800 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.455493 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.455642 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.459554 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:48.459562 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:48.459568 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:48.459623 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:48.463847 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.464713 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.464719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.464722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.464854 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.465335 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.465342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.465347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.465807 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:48.465970 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:48.468263 (+ 2293us) server_negotiation.cc:299] Negotiation successful
0808 13:02:48.468497 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":439,"thread_start_us":324,"threads_started":1}
I20260808 13:02:48.472723 27075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } has no permanent_uuid. Determining permanent_uuid...
I20260808 13:02:48.475749 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.459228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47932)
0808 13:02:48.459514 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.459545 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:48.459678 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:48.460592 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:48.460597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:48.461044 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:48.461195 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:48.461204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.463076 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.463082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:48.464218 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:48.464235 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.465245 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.465253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:48.465492 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.466079 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:48.466095 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:48.468942 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:48.472831 (+ 3889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:48.472835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:48.472839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:48.473442 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:48.473909 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:48.473915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:48.473918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:48.474093 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:48.474757 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:48.474762 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:48.474915 (+ 153us) client_negotiation.cc:769] Sending connection context
0808 13:02:48.475067 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 13:02:48.475348 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":3}
I20260808 13:02:48.475780 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.459274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.24.167.60:38777)
0808 13:02:48.460297 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.460302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.460326 (+ 24us) server_negotiation.cc:407] Connection header received
0808 13:02:48.460371 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.460376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.460452 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.460556 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.461366 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.462755 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.464378 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.465068 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.465198 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.469214 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:48.469220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:48.469225 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:48.469258 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:48.472666 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.473609 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.473617 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.473625 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.473774 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.474217 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.474222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.474225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.474607 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:48.474779 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:48.475171 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 13:02:48.475372 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":692,"threads_started":1}
Aug 08 13:02:48 dist-test-slave-tk9z krb5kdc[26771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
Aug 08 13:02:48 dist-test-slave-tk9z krb5kdc[26771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194168, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.60@KRBTEST.COM for kudu/127.24.167.61@KRBTEST.COM
I20260808 13:02:48.482815 26869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917: Bootstrap starting.
I20260808 13:02:48.485566 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.466469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45198 (local address 127.24.167.62:46549)
0808 13:02:48.467098 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.467102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.467124 (+ 22us) server_negotiation.cc:407] Connection header received
0808 13:02:48.467164 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.467167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.467222 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.467305 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.469003 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.470595 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.472466 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.472939 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.473060 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.478317 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:48.478324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:48.478330 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:48.478394 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:48.481980 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.482882 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.482887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.482891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.483069 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.483586 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.483590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.483592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.484023 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:48.484296 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:48.484883 (+ 587us) server_negotiation.cc:299] Negotiation successful
0808 13:02:48.485119 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":339,"threads_started":1}
I20260808 13:02:48.487197 26869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:48.489470 26869 log.cc:824] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:48.490118 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.473563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.24.167.61:36121)
0808 13:02:48.473775 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.473783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.473898 (+ 115us) server_negotiation.cc:407] Connection header received
0808 13:02:48.474116 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.474122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.474225 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.474332 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.475354 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.476497 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.478001 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.478695 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.478819 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.482792 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:48.482802 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:48.482807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:48.482855 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:48.486595 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.487637 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.487643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.487646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.487867 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.488298 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.488303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.488307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.488667 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:48.488829 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:48.489433 (+ 604us) server_negotiation.cc:299] Negotiation successful
0808 13:02:48.489698 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 13:02:48.491111 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.473560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:36121 (local address 127.0.0.1:33518)
0808 13:02:48.473814 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.473834 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:48.473986 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:48.474413 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:48.474419 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:48.474933 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:48.475162 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:48.475175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.476695 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.476700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:48.477852 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:48.477865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.479063 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.479072 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:48.479254 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.479787 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:48.479814 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:48.482542 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:48.486795 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:48.486800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:48.486804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:48.487456 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:48.488009 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:48.488014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:48.488018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:48.488180 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:48.488921 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:48.488927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:48.489119 (+ 192us) client_negotiation.cc:769] Sending connection context
0808 13:02:48.490430 (+ 1311us) client_negotiation.cc:240] Negotiation successful
0808 13:02:48.490760 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260808 13:02:48.492483 27075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } has no permanent_uuid. Determining permanent_uuid...
I20260808 13:02:48.493829 26869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917: No bootstrap required, opened a new log
Aug 08 13:02:48 dist-test-slave-tk9z krb5kdc[26771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.61@KRBTEST.COM
Aug 08 13:02:48 dist-test-slave-tk9z krb5kdc[26771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194168, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.60@KRBTEST.COM for kudu/127.24.167.60@KRBTEST.COM
I20260808 13:02:48.504240 26869 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } }
I20260808 13:02:48.504607 26869 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d3aaf594294ae08c9dafd0cb46e917, State: Initialized, Role: FOLLOWER
I20260808 13:02:48.505178 26869 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } }
I20260808 13:02:48.507105 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.487567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33520 (local address 127.24.167.61:36121)
0808 13:02:48.488374 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.488381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.488407 (+ 26us) server_negotiation.cc:407] Connection header received
0808 13:02:48.488483 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.488490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.488572 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.488695 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.490508 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.491696 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.493562 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.494160 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.494260 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.499980 (+ 5720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:48.499988 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:48.499993 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:48.500036 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:48.503777 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.504685 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.504690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.504694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.504859 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.505314 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.505319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.505323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.505623 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:48.505916 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:48.506301 (+ 385us) server_negotiation.cc:299] Negotiation successful
0808 13:02:48.506666 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":431,"threads_started":1}
I20260808 13:02:48.507781 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.493219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47940)
0808 13:02:48.493386 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.493405 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:48.493515 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:48.493922 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:48.493928 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:48.494453 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:48.494702 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:48.494715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.496040 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.496049 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:48.497559 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:48.497574 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.498566 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.498573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:48.498808 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.499436 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:48.499461 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:48.501540 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:48.505014 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:48.505019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:48.505022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:48.505575 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:48.505997 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:48.506000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:48.506006 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:48.506123 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:48.506804 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:48.506808 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:48.506966 (+ 158us) client_negotiation.cc:769] Sending connection context
0808 13:02:48.507115 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 13:02:48.507434 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 13:02:48.508495 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.493459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.24.167.60:38777)
0808 13:02:48.493605 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.493611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.493628 (+ 17us) server_negotiation.cc:407] Connection header received
0808 13:02:48.493675 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.493680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.493764 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.493873 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.494860 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.495877 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.497720 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.498344 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.498534 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.501724 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:48.501728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:48.501731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:48.501758 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:48.504880 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.505704 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.505710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.505716 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.505865 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.506251 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.506255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.506258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.506608 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:48.506723 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:48.507997 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0808 13:02:48.508153 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 13:02:48.510972 27090 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } } }
I20260808 13:02:48.511474 27090 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:02:48.512584 26869 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:48.512930 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:48.514804 27075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a: Bootstrap starting.
I20260808 13:02:48.517330 26839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a4d3aaf594294ae08c9dafd0cb46e917"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45198
I20260808 13:02:48.517866 26839 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 13:02:48.518179 27075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:48.518162 26839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:48.519323 27075 log.cc:824] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:48.521138 26839 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } }
I20260808 13:02:48.522701 27075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a: No bootstrap required, opened a new log
I20260808 13:02:48.523080 26839 leader_election.cc:288] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [CANDIDATE]: Term 1 election: Requested vote from peers f4d647d20e5a4caea73e963ab877c1f6 (127.24.167.61:36121), 77907cb6a96e4926a062b67d418a464a (127.24.167.60:38777)
I20260808 13:02:48.538148 27075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } }
I20260808 13:02:48.538842 27075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77907cb6a96e4926a062b67d418a464a, State: Initialized, Role: FOLLOWER
I20260808 13:02:48.539615 27075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } }
I20260808 13:02:48.542548 27075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:48.543090 27034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "77907cb6a96e4926a062b67d418a464a"
W20260808 13:02:48.544147 26817 tablet.cc:2369] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:02:48.544389 27034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:48.548919 27034 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4d3aaf594294ae08c9dafd0cb46e917 in term 1.
I20260808 13:02:48.549706 26811 leader_election.cc:302] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77907cb6a96e4926a062b67d418a464a, a4d3aaf594294ae08c9dafd0cb46e917; no voters:
I20260808 13:02:48.550302 27090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:48.541517 27104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } } }
I20260808 13:02:48.556685 27104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:02:48.562451 27090 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [term 1 LEADER]: Becoming Leader. State: Replica: a4d3aaf594294ae08c9dafd0cb46e917, State: Running, Role: LEADER
I20260808 13:02:48.563248 27090 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } }
I20260808 13:02:48.585006 27114 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4d3aaf594294ae08c9dafd0cb46e917. Latest consensus state: current_term: 1 leader_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } } }
I20260808 13:02:48.586046 27114 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:48.599839 26970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } attempt: 5
I20260808 13:02:48.612061 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.542714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33530 (local address 127.24.167.61:36121)
0808 13:02:48.542914 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.542920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.548774 (+ 5854us) server_negotiation.cc:407] Connection header received
0808 13:02:48.567052 (+ 18278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.567063 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.567244 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.567343 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.578888 (+ 11545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.579998 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.587835 (+ 7837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.588789 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.588920 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.596380 (+ 7460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:48.596393 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:48.596401 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:48.596459 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:48.605740 (+ 9281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.606614 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.606620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.606624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.606779 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.607938 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.607943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.607946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.608324 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:48.611244 (+ 2920us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:48.611423 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 13:02:48.611671 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 13:02:48.613186 27120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 08 13:02:48 dist-test-slave-tk9z krb5kdc[26771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194167, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.61@KRBTEST.COM for kudu/127.24.167.60@KRBTEST.COM
I20260808 13:02:48.614302 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.542724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45204 (local address 127.24.167.62:46549)
0808 13:02:48.542914 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.542920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.571820 (+ 28900us) server_negotiation.cc:407] Connection header received
0808 13:02:48.572220 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.572228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.572334 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.572439 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.582158 (+ 9719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.583529 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.585569 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.586315 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.586531 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.590889 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:48.590895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:48.590900 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:48.590957 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:48.604727 (+ 13770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.606943 (+ 2216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.606949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.606952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.607132 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.608483 (+ 1351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.608488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.608491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.608889 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:48.609165 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:48.613679 (+ 4514us) server_negotiation.cc:299] Negotiation successful
0808 13:02:48.613944 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 13:02:48.618712 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.550282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.24.167.60:38777)
0808 13:02:48.550526 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.550535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.575130 (+ 24595us) server_negotiation.cc:407] Connection header received
0808 13:02:48.575312 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.575321 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.575431 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.575570 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.578082 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.579678 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.584751 (+ 5073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.585805 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.585951 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.595637 (+ 9686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:48.595646 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:48.595652 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:48.595706 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:48.601704 (+ 5998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.614480 (+ 12776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.614489 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.614495 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.614731 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.615274 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.615282 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.615288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.615739 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:48.616113 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:48.617053 (+ 940us) server_negotiation.cc:299] Negotiation successful
0808 13:02:48.617492 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 13:02:48.597157 26789 master_runner.cc:186] Error getting master registration for 127.24.167.61:36121: OK, instance_id { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" instance_seqno: 1786194167866063 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 13:02:48.624331 26789 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 13:02:48.630316 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.602257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.24.167.60:38777)
0808 13:02:48.602500 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.602509 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.602533 (+ 24us) server_negotiation.cc:407] Connection header received
0808 13:02:48.602604 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.602612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.602707 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.602826 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.604326 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.605763 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.608208 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.609148 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.609304 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.617305 (+ 8001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:48.617315 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:48.617322 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:48.617372 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:48.624269 (+ 6897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.626857 (+ 2588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.626866 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.626872 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.627089 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:48.628034 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:48.628042 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:48.628047 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:48.628525 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:48.628704 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:48.629423 (+ 719us) server_negotiation.cc:299] Negotiation successful
0808 13:02:48.629783 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 13:02:48.631680 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.600432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:38777 (local address 127.0.0.1:47968)
0808 13:02:48.600711 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:48.600731 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:48.600848 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:48.603109 (+ 2261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:48.603118 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:48.603797 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:48.604094 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:48.604110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.605984 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.605993 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:48.607972 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:48.607997 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.609550 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.609559 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:48.609879 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.610710 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:48.610742 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:48.617051 (+ 6309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:48.624554 (+ 7503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:48.624563 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:48.624607 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:48.626589 (+ 1982us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:48.627300 (+ 711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:48.627356 (+ 56us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:48.627365 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:48.627844 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:48.628792 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:48.628800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:48.629047 (+ 247us) client_negotiation.cc:769] Sending connection context
0808 13:02:48.630680 (+ 1633us) client_negotiation.cc:240] Negotiation successful
0808 13:02:48.631130 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260808 13:02:48.634054 27120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:48.637945 26970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6: Bootstrap starting.
I20260808 13:02:48.642011 26970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:48.643462 26970 log.cc:824] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:48.651665 27034 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a [term 1 FOLLOWER]: Refusing update from remote peer a4d3aaf594294ae08c9dafd0cb46e917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:02:48.652848 27114 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:02:48.659030 26970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6: No bootstrap required, opened a new log
W20260808 13:02:48.662285 27004 tablet.cc:2369] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:02:48.678445 27104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4d3aaf594294ae08c9dafd0cb46e917. Latest consensus state: current_term: 1 leader_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } } }
I20260808 13:02:48.678814 27104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:02:48.681252 27131 mvcc.cc:205] Tried to move back new op lower bound from 7316251314780999680 to 7316251314443657216. Current Snapshot: MvccSnapshot[applied={T|T < 7316251314780999680}]
I20260808 13:02:48.686467 27140 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 13:02:48.688131 27136 mvcc.cc:205] Tried to move back new op lower bound from 7316251314780999680 to 7316251314443657216. Current Snapshot: MvccSnapshot[applied={T|T < 7316251314780999680}]
I20260808 13:02:48.689404 27104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } } }
I20260808 13:02:48.689695 27104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:02:48.693959 27120 catalog_manager.cc:1347] Generated new cluster ID: 32cf059471974720ac00925eac17b4d3
I20260808 13:02:48.694077 27120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:48.695355 27114 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } } }
I20260808 13:02:48.695693 27114 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:48.697399 26970 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } }
I20260808 13:02:48.698136 26970 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d647d20e5a4caea73e963ab877c1f6, State: Initialized, Role: FOLLOWER
I20260808 13:02:48.698740 26970 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } }
I20260808 13:02:48.700156 27146 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } } }
I20260808 13:02:48.701050 27146 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:02:48.701274 26970 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:48.701570 26936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f4d647d20e5a4caea73e963ab877c1f6"
I20260808 13:02:48.701867 26935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:48.705694 26935 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6 [term 1 FOLLOWER]: Refusing update from remote peer a4d3aaf594294ae08c9dafd0cb46e917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:02:48.709004 26936 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4d3aaf594294ae08c9dafd0cb46e917 in term 1.
I20260808 13:02:48.699556 26979 master_runner.cc:186] Error getting master registration for 127.24.167.61:36121: OK, instance_id { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" instance_seqno: 1786194167866063 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 13:02:48.712172 27090 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:02:48.718640 26979 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 13:02:48.734316 27120 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:48.736357 27120 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:48.743682 27114 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } } }
I20260808 13:02:48.744009 27114 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:48.746583 27146 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4d3aaf594294ae08c9dafd0cb46e917. Latest consensus state: current_term: 1 leader_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } } }
I20260808 13:02:48.746935 27146 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:02:48.746740 27153 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d3aaf594294ae08c9dafd0cb46e917" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46549 } } peers { permanent_uuid: "f4d647d20e5a4caea73e963ab877c1f6" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 36121 } } peers { permanent_uuid: "77907cb6a96e4926a062b67d418a464a" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 38777 } } }
I20260808 13:02:48.751219 27153 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:02:48.754223 27120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4d3aaf594294ae08c9dafd0cb46e917: Generated new TSK 0
I20260808 13:02:48.755527 27120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:48.794993 27168 catalog_manager.cc:1259] Loaded cluster ID: 32cf059471974720ac00925eac17b4d3
I20260808 13:02:48.795174 27168 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6: loading cluster ID for follower catalog manager: success
I20260808 13:02:48.799659 27168 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6: acquiring CA information for follower catalog manager: success
I20260808 13:02:48.802716 27168 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f4d647d20e5a4caea73e963ab877c1f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 13:02:48.857337 27083 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:48.844226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45220 (local address 127.24.167.62:46549)
0808 13:02:48.844779 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.844789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.853890 (+ 9101us) server_negotiation.cc:407] Connection header received
0808 13:02:48.855037 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.855044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.855141 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.855236 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.856897 (+ 1661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45220: BlockingRecv error: recv got EOF from 127.0.0.1:45220 (error 108)
Metrics: {"server-negotiator.queue_time_us":363}
W20260808 13:02:48.862296 27087 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:48.844501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.24.167.61:36121)
0808 13:02:48.844981 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.844990 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.859342 (+ 14352us) server_negotiation.cc:407] Connection header received
0808 13:02:48.860320 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.860330 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.860469 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.860565 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.861933 (+ 1368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33532: BlockingRecv error: recv got EOF from 127.0.0.1:33532 (error 108)
Metrics: {"server-negotiator.queue_time_us":313}
W20260808 13:02:48.871798 27079 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:48.854180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47970 (local address 127.24.167.60:38777)
0808 13:02:48.854427 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.854433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.864702 (+ 10269us) server_negotiation.cc:407] Connection header received
0808 13:02:48.865874 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.865883 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.866012 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.866104 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.871499 (+ 5395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47970: BlockingRecv error: recv got EOF from 127.0.0.1:47970 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 13:02:48.894531 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:48.876270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.24.167.61:36121)
0808 13:02:48.876590 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.876597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.886494 (+ 9897us) server_negotiation.cc:407] Connection header received
0808 13:02:48.886670 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.886679 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.886961 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.887161 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:48.887695 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.888939 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.891073 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:48.891826 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:48.891925 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:48.892803 (+ 878us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:48.893579 (+ 776us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:48.893704 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:48.894019 (+ 315us) server_negotiation.cc:299] Negotiation successful
0808 13:02:48.894127 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
W20260808 13:02:48.967417 27079 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:48.956152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47986 (local address 127.24.167.60:38777)
0808 13:02:48.956349 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:48.956355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:48.965314 (+ 8959us) server_negotiation.cc:407] Connection header received
0808 13:02:48.965641 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:48.965648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:48.965812 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:48.965910 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:48.967151 (+ 1241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47986: BlockingRecv error: recv got EOF from 127.0.0.1:47986 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 13:02:49.019183 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:49.002215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33546 (local address 127.24.167.61:36121)
0808 13:02:49.002453 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.002461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.009935 (+ 7474us) server_negotiation.cc:407] Connection header received
0808 13:02:49.010584 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.010591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.010689 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.010763 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:49.011547 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.014497 (+ 2950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.016688 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.017444 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.017553 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:49.018231 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:49.018431 (+ 200us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:49.018500 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:49.018734 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 13:02:49.018829 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260808 13:02:49.067174 27079 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:49.053692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.24.167.60:38777)
0808 13:02:49.053892 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.053899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.065158 (+ 11259us) server_negotiation.cc:407] Connection header received
0808 13:02:49.065513 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.065520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.065633 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.065715 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:49.066908 (+ 1193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47994: BlockingRecv error: recv got EOF from 127.0.0.1:47994 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 13:02:49.132902 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:49.117521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33550 (local address 127.24.167.61:36121)
0808 13:02:49.117749 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.117756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.125649 (+ 7893us) server_negotiation.cc:407] Connection header received
0808 13:02:49.126213 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.126220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.126317 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.126407 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:49.127134 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.128321 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.130496 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.131187 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.131291 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:49.131946 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:49.132143 (+ 197us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:49.132212 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:49.132464 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 13:02:49.132561 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260808 13:02:49.182415 27079 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:49.169231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48010 (local address 127.24.167.60:38777)
0808 13:02:49.169447 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.169456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.180275 (+ 10819us) server_negotiation.cc:407] Connection header received
0808 13:02:49.180830 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.180836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.180945 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.181026 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:49.182154 (+ 1128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48010: BlockingRecv error: recv got EOF from 127.0.0.1:48010 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 13:02:49.244626 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:49.229400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.24.167.61:36121)
0808 13:02:49.229633 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.229641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.237193 (+ 7552us) server_negotiation.cc:407] Connection header received
0808 13:02:49.237774 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.237782 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.237894 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.237973 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:49.238769 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.240000 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.242029 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.242755 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.242873 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:49.243639 (+ 766us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:49.243806 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:49.243869 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:49.244179 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 13:02:49.244284 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260808 13:02:49.296362 27079 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:49.284003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48012 (local address 127.24.167.60:38777)
0808 13:02:49.284248 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.284255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.293923 (+ 9668us) server_negotiation.cc:407] Connection header received
0808 13:02:49.294781 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.294788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.294898 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.294985 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:49.296111 (+ 1126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48012: BlockingRecv error: recv got EOF from 127.0.0.1:48012 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
I20260808 13:02:49.372617 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:49.350286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.24.167.61:36121)
0808 13:02:49.350519 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.350526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.363875 (+ 13349us) server_negotiation.cc:407] Connection header received
0808 13:02:49.364287 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.364293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.364449 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.364526 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:49.366301 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.367551 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.369708 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.370496 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.370640 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:49.371446 (+ 806us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:49.371627 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:49.371698 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:49.372129 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 13:02:49.372245 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260808 13:02:49.385699 27079 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:49.383325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48024 (local address 127.24.167.60:38777)
0808 13:02:49.383582 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.383592 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.384172 (+ 580us) server_negotiation.cc:407] Connection header received
0808 13:02:49.384406 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.384412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.384488 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.384603 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:49.385409 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48024: BlockingRecv error: recv got EOF from 127.0.0.1:48024 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 13:02:49.411209 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:49.405202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.24.167.61:36121)
0808 13:02:49.405464 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.405474 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.406054 (+ 580us) server_negotiation.cc:407] Connection header received
0808 13:02:49.406306 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.406310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.406433 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.406560 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:49.407052 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.407848 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.409251 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.409731 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.409879 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:49.410325 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:49.410520 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:49.410635 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:49.410792 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 13:02:49.410878 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260808 13:02:49.420027 27079 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:49.417587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.24.167.60:38777)
0808 13:02:49.417832 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.417842 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.418458 (+ 616us) server_negotiation.cc:407] Connection header received
0808 13:02:49.418712 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.418718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.418808 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.418923 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:49.419739 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48038: BlockingRecv error: recv got EOF from 127.0.0.1:48038 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 13:02:49.461666 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:49.455887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.24.167.61:36121)
0808 13:02:49.456124 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.456134 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.456693 (+ 559us) server_negotiation.cc:407] Connection header received
0808 13:02:49.456944 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.456949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.457021 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.457127 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:49.457572 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.458450 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.459780 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.460246 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.460397 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:49.460840 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:49.460974 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:49.461081 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:49.461242 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 13:02:49.461326 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260808 13:02:49.472621 27079 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:49.468647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48054 (local address 127.24.167.60:38777)
0808 13:02:49.468853 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.468860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.471007 (+ 2147us) server_negotiation.cc:407] Connection header received
0808 13:02:49.471294 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.471299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.471394 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.471537 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:49.472397 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48054: BlockingRecv error: recv got EOF from 127.0.0.1:48054 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 13:02:49.545981 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:49.540329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.24.167.61:36121)
0808 13:02:49.540654 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.540662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.541018 (+ 356us) server_negotiation.cc:407] Connection header received
0808 13:02:49.541252 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.541257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.541328 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.541435 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:49.541903 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.542762 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.544194 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.544647 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.544791 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:49.545194 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:49.545316 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:49.545420 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:49.545593 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 13:02:49.545677 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260808 13:02:49.554437 27079 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:49.552389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48060 (local address 127.24.167.60:38777)
0808 13:02:49.552648 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.552655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.552978 (+ 323us) server_negotiation.cc:407] Connection header received
0808 13:02:49.553195 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.553201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.553291 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.553404 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:49.554186 (+ 782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48060: BlockingRecv error: recv got EOF from 127.0.0.1:48060 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 13:02:49.691061 27140 catalog_manager.cc:1259] Loaded cluster ID: 32cf059471974720ac00925eac17b4d3
I20260808 13:02:49.691217 27140 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a: loading cluster ID for follower catalog manager: success
I20260808 13:02:49.692453 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:49.686236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.24.167.61:36121)
0808 13:02:49.686520 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.686527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.686788 (+ 261us) server_negotiation.cc:407] Connection header received
0808 13:02:49.687048 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.687052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.687124 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.687233 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:49.687718 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.688439 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.690490 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.690931 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.691083 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:49.691599 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:49.691744 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:49.691846 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:49.692039 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 13:02:49.692153 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260808 13:02:49.695256 27140 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a: acquiring CA information for follower catalog manager: success
I20260808 13:02:49.697129 27140 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 77907cb6a96e4926a062b67d418a464a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 13:02:49.706015 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:49.699669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48066 (local address 127.24.167.60:38777)
0808 13:02:49.699890 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.699896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.700207 (+ 311us) server_negotiation.cc:407] Connection header received
0808 13:02:49.700383 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.700386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.700534 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.700628 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:49.701107 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.701891 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.703850 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.704256 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.704354 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:49.704903 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:49.705347 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:49.705485 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:49.705625 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 13:02:49.705699 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 13:02:49.725703 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:49.714464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48070 (local address 127.24.167.60:38777)
0808 13:02:49.714809 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.714815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.719057 (+ 4242us) server_negotiation.cc:407] Connection header received
0808 13:02:49.719252 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.719261 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.719369 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.719477 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:49.720416 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.721289 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.723702 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.724176 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.724339 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:49.724865 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:49.724975 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:49.725094 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:49.725235 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 13:02:49.725359 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260808 13:02:49.729959 27321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:49.712919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45236 (local address 127.24.167.62:46549)
0808 13:02:49.713526 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.713533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.719526 (+ 5993us) server_negotiation.cc:407] Connection header received
0808 13:02:49.719663 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.719667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.719948 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.720133 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:49.721054 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.722272 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.725737 (+ 3465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.726491 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.726622 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:49.728788 (+ 2166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:49.729241 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:49.729353 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:49.729497 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 13:02:49.729603 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":275,"threads_started":1}
I20260808 13:02:49.730348 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:49.714262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33604 (local address 127.24.167.61:36121)
0808 13:02:49.714661 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:49.714667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:49.719718 (+ 5051us) server_negotiation.cc:407] Connection header received
0808 13:02:49.719896 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:49.719901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:49.720012 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:49.720115 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:49.721658 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.722746 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.728211 (+ 5465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:49.728680 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:49.728817 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:49.729329 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:49.729508 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:49.729596 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:49.729925 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 13:02:49.730022 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260808 13:02:49.736487 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26789
I20260808 13:02:49.737582 26862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:49.898906 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 26789
W20260808 13:02:49.932473 26999 connection.cc:535] server connection from 127.0.0.1:47932 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 13:02:49.932960 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26878
I20260808 13:02:49.934197 26965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:50.046022 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 26878
I20260808 13:02:50.074337 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 26979
I20260808 13:02:50.075685 27067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:50.197240 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 26979
2026-08-08T13:02:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5051 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 13:02:50 dist-test-slave-tk9z krb5kdc[27332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 13:02:50 dist-test-slave-tk9z krb5kdc[27332](info): set up 2 sockets
Aug 08 13:02:50 dist-test-slave-tk9z krb5kdc[27332](info): commencing operation
krb5kdc: starting...
W20260808 13:02:51.566740 25244 mini_kdc.cc:121] Time spent starting KDC: real 1.320s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/test-user.keytab.
Aug 08 13:02:51 dist-test-slave-tk9z krb5kdc[27332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T13:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T13:02:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.167.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
Entry for principal kudu/127.24.167.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab.
I20260808 13:02:51.921773 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:46349
--webserver_interface=127.24.167.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:41167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:46349,127.24.167.61:40567,127.24.167.60:39503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 13:02:52.290081 27348 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 13:02:52.290515 27348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:52.290645 27348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:52.290787 27348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:52.298282 27348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:52.298442 27348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:52.298569 27348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:52.298666 27348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:52.298758 27348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:52.309702 27348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:41167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:46349,127.24.167.61:40567,127.24.167.60:39503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.62.keytab
--principal=kudu/127.24.167.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.62:46349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.167.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:52.311183 27348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:52.312522 27348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:52.324572 27356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:52.324867 27357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:52.324872 27360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:52.325987 27348 server_base.cc:1034] running on GCE node
I20260808 13:02:52.326975 27348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:52.328668 27348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:52.329967 27348 hybrid_clock.cc:648] HybridClock initialized: now 1786194172329896 us; error 63 us; skew 500 ppm
I20260808 13:02:52.330440 27348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:52 dist-test-slave-tk9z krb5kdc[27332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194172, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:52.334201 27348 init.cc:377] Logged in from keytab as kudu/127.24.167.62@KRBTEST.COM (short username kudu)
I20260808 13:02:52.335938 27348 webserver.cc:460] Webserver started at http://127.24.167.62:45645/ using document root <none> and password file <none>
I20260808 13:02:52.336714 27348 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:52.336844 27348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:52.337214 27348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:52.340629 27348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/data/instance:
uuid: "9f7dee8229a34e01ae6601c581475265"
format_stamp: "Formatted at 2026-08-08 13:02:52 on dist-test-slave-tk9z"
server_key: "18d528ac8758e62de5a0cc66a102e991"
server_key_iv: "1ffdf7ee86e489c497e911b92266260d"
server_key_version: "encryptionkey@0"
I20260808 13:02:52.341574 27348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/wal/instance:
uuid: "9f7dee8229a34e01ae6601c581475265"
format_stamp: "Formatted at 2026-08-08 13:02:52 on dist-test-slave-tk9z"
server_key: "18d528ac8758e62de5a0cc66a102e991"
server_key_iv: "1ffdf7ee86e489c497e911b92266260d"
server_key_version: "encryptionkey@0"
I20260808 13:02:52.347506 27348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260808 13:02:52.351343 27368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:52.352829 27348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 13:02:52.353048 27348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/wal
uuid: "9f7dee8229a34e01ae6601c581475265"
format_stamp: "Formatted at 2026-08-08 13:02:52 on dist-test-slave-tk9z"
server_key: "18d528ac8758e62de5a0cc66a102e991"
server_key_iv: "1ffdf7ee86e489c497e911b92266260d"
server_key_version: "encryptionkey@0"
I20260808 13:02:52.353281 27348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:52.382639 27348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:52.387444 27348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:52.387758 27348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:52.407711 27348 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.62:46349
I20260808 13:02:52.407685 27432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.62:46349 every 8 connection(s)
I20260808 13:02:52.409077 27348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/data/info.pb
I20260808 13:02:52.412784 27433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:52.417742 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 27348
I20260808 13:02:52.417936 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-0/wal/instance
I20260808 13:02:52.418462 25244 external_mini_cluster.cc:1428] Setting key 32ff0286ad72cc07cf8ae64c8b28c3bb
I20260808 13:02:52.419539 27433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 13:02:52 dist-test-slave-tk9z krb5kdc[27332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194172, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:02:52.442584 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.422642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.24.167.62:46349)
0808 13:02:52.423477 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:52.423496 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:52.423897 (+ 401us) server_negotiation.cc:407] Connection header received
0808 13:02:52.425878 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:52.425884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:52.426094 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:52.426296 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:52.427590 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:52.428768 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:52.430131 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:52.430740 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:52.430941 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:52.435343 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:52.435351 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:52.435362 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:52.435395 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:52.438135 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:52.439141 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:52.439148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:52.439150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:52.439331 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:52.439825 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:52.439828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:52.439830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:52.440329 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:52.440477 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:52.441227 (+ 750us) server_negotiation.cc:299] Negotiation successful
0808 13:02:52.441512 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":579,"thread_start_us":361,"threads_started":1}
I20260808 13:02:52.442642 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.422665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:46349 (local address 127.0.0.1:34690)
0808 13:02:52.423746 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:52.423778 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:52.425525 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:52.426319 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:52.426325 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:52.427052 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:52.427381 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:52.427397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:52.428938 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:52.428941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:52.429989 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:52.430001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:52.430980 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:52.430988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:52.431225 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:52.431897 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:52.431934 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:52.435122 (+ 3188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:52.438274 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:52.438289 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:52.438303 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:52.438932 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:52.439474 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:52.439482 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:52.439488 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:52.439710 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:52.440478 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:52.440488 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:52.440777 (+ 289us) client_negotiation.cc:769] Sending connection context
0808 13:02:52.441095 (+ 318us) client_negotiation.cc:240] Negotiation successful
0808 13:02:52.441711 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":116,"thread_start_us":698,"threads_started":1}
I20260808 13:02:52.445611 27433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } has no permanent_uuid. Determining permanent_uuid...
I20260808 13:02:52.446899 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.446353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:40567 (local address 127.0.0.1:39944)
0808 13:02:52.446578 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:52.446717 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260808 13:02:52.447230 27372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
I20260808 13:02:52.448968 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.448471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:40567 (local address 127.0.0.1:39958)
0808 13:02:52.448662 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:52.448767 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 13:02:52.449213 27433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.61:40567: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.24.167.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.61.keytab.
Entry for principal kudu/127.24.167.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.61.keytab.
I20260808 13:02:52.491027 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.61:40567
--webserver_interface=127.24.167.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.61.keytab
--principal=kudu/127.24.167.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:41167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:46349,127.24.167.61:40567,127.24.167.60:39503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 13:02:52.498570 27433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } attempt: 1
I20260808 13:02:52.499940 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.499406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:40567 (local address 127.0.0.1:39962)
0808 13:02:52.499640 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:52.499796 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260808 13:02:52.501140 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.500688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:40567 (local address 127.0.0.1:39974)
0808 13:02:52.500870 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:52.500987 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260808 13:02:52.501332 27433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.61:40567: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
I20260808 13:02:52.569592 27433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } attempt: 2
I20260808 13:02:52.570932 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.570325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:40567 (local address 127.0.0.1:39990)
0808 13:02:52.570668 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:52.570763 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260808 13:02:52.571965 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.571584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:40567 (local address 127.0.0.1:40006)
0808 13:02:52.571730 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:52.571819 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 13:02:52.572225 27433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.61:40567: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
I20260808 13:02:52.663497 27433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } attempt: 3
I20260808 13:02:52.664860 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.664353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:40567 (local address 127.0.0.1:40008)
0808 13:02:52.664577 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:52.664684 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 13:02:52.666059 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.665664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:40567 (local address 127.0.0.1:40012)
0808 13:02:52.665813 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:52.665902 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 13:02:52.666349 27433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.61:40567: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
I20260808 13:02:52.809783 27433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } attempt: 4
I20260808 13:02:52.811210 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.810622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:40567 (local address 127.0.0.1:40022)
0808 13:02:52.810956 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:52.811053 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260808 13:02:52.812240 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.811888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:40567 (local address 127.0.0.1:40030)
0808 13:02:52.812028 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:52.812110 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 13:02:52.812515 27433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.61:40567: Network error: Client connection negotiation failed: client connection to 127.24.167.61:40567: connect: Connection refused (error 111)
W20260808 13:02:52.840965 27444 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 13:02:52.841286 27444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:52.841449 27444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:52.841579 27444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:52.848783 27444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:52.848876 27444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:52.848968 27444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:52.849048 27444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:52.849138 27444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:52.859256 27444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:41167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:46349,127.24.167.61:40567,127.24.167.60:39503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.61.keytab
--principal=kudu/127.24.167.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.61:40567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.167.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:52.860567 27444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:52.861912 27444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:52.873744 27451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:52.873860 27452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:52.873785 27454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:52.879873 27444 server_base.cc:1034] running on GCE node
I20260808 13:02:52.880761 27444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:52.882285 27444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:52.883554 27444 hybrid_clock.cc:648] HybridClock initialized: now 1786194172883486 us; error 33 us; skew 500 ppm
I20260808 13:02:52.883891 27444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:52 dist-test-slave-tk9z krb5kdc[27332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194172, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:52.887109 27444 init.cc:377] Logged in from keytab as kudu/127.24.167.61@KRBTEST.COM (short username kudu)
I20260808 13:02:52.888571 27444 webserver.cc:460] Webserver started at http://127.24.167.61:34407/ using document root <none> and password file <none>
I20260808 13:02:52.889302 27444 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:52.889433 27444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:52.889761 27444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:52.892970 27444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/data/instance:
uuid: "8200805955f1414a8976796841bdedcf"
format_stamp: "Formatted at 2026-08-08 13:02:52 on dist-test-slave-tk9z"
server_key: "3b87b159c585ac53e83107233379c201"
server_key_iv: "b7d110220f8aefc9468f152b24d150fa"
server_key_version: "encryptionkey@0"
I20260808 13:02:52.893805 27444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/wal/instance:
uuid: "8200805955f1414a8976796841bdedcf"
format_stamp: "Formatted at 2026-08-08 13:02:52 on dist-test-slave-tk9z"
server_key: "3b87b159c585ac53e83107233379c201"
server_key_iv: "b7d110220f8aefc9468f152b24d150fa"
server_key_version: "encryptionkey@0"
I20260808 13:02:52.899231 27444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 13:02:52.902731 27462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:52.904052 27444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 13:02:52.904239 27444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/wal
uuid: "8200805955f1414a8976796841bdedcf"
format_stamp: "Formatted at 2026-08-08 13:02:52 on dist-test-slave-tk9z"
server_key: "3b87b159c585ac53e83107233379c201"
server_key_iv: "b7d110220f8aefc9468f152b24d150fa"
server_key_version: "encryptionkey@0"
I20260808 13:02:52.904457 27444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:52.923107 27444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:52.927196 27444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:52.927484 27444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:52.946835 27444 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.61:40567
I20260808 13:02:52.946872 27528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.61:40567 every 8 connection(s)
I20260808 13:02:52.948294 27444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/data/info.pb
I20260808 13:02:52.952229 27529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:52.954006 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 27444
I20260808 13:02:52.954223 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-1/wal/instance
I20260808 13:02:52.954725 25244 external_mini_cluster.cc:1428] Setting key 11ad9b73efaf8679c21b2d091953e82b
I20260808 13:02:52.959430 27529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 13:02:52 dist-test-slave-tk9z krb5kdc[27332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194172, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.61@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:02:52.984097 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.962093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.24.167.62:46349)
0808 13:02:52.962695 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:52.962702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:52.963686 (+ 984us) server_negotiation.cc:407] Connection header received
0808 13:02:52.966706 (+ 3020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:52.966712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:52.966802 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:52.966915 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:52.969138 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:52.970412 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:52.971940 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:52.972720 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:52.972886 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:52.976423 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:52.976432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:52.976440 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:52.976500 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:52.980454 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:52.981339 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:52.981347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:52.981353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:52.981518 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:52.981929 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:52.981937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:52.981943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:52.982428 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:52.982600 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:52.983301 (+ 701us) server_negotiation.cc:299] Negotiation successful
0808 13:02:52.983570 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":308,"threads_started":1}
I20260808 13:02:52.984932 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.962680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:46349 (local address 127.0.0.1:34700)
0808 13:02:52.963501 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:52.963563 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:52.966342 (+ 2779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:52.967197 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:52.967210 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:52.968059 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:52.968880 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:52.968909 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:52.970568 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:52.970572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:52.971778 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:52.971790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:52.972961 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:52.972967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:52.973172 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:52.973873 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:52.973902 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:52.976173 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:52.980687 (+ 4514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:52.980694 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:52.980708 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:52.981203 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:52.981630 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:52.981633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:52.981636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:52.981815 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:52.982549 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:52.982556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:52.982902 (+ 346us) client_negotiation.cc:769] Sending connection context
0808 13:02:52.983158 (+ 256us) client_negotiation.cc:240] Negotiation successful
0808 13:02:52.983562 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":290,"threads_started":1}
I20260808 13:02:52.986919 27529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.167.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.167.60@KRBTEST.COM" created.
Aug 08 13:02:52 dist-test-slave-tk9z krb5kdc[27332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194172, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.61@KRBTEST.COM for kudu/127.24.167.61@KRBTEST.COM
I20260808 13:02:53.004084 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.987810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:40567 (local address 127.0.0.1:40032)
0808 13:02:52.988058 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:52.988080 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:52.988195 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:52.989157 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:52.989164 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:52.989596 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:52.989741 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:52.989749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:52.991389 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:52.991393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:52.992550 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:52.992563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:52.993812 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:52.993817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:52.993975 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:52.994605 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:52.994622 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:52.996906 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:53.001376 (+ 4470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:53.001380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:53.001383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:53.001776 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:53.002308 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:53.002312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:53.002314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:53.002509 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:53.003327 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:53.003331 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:53.003480 (+ 149us) client_negotiation.cc:769] Sending connection context
0808 13:02:53.003594 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 13:02:53.003806 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 13:02:53.004768 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:52.987986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.24.167.61:40567)
0808 13:02:52.988728 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:52.988734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:52.988759 (+ 25us) server_negotiation.cc:407] Connection header received
0808 13:02:52.988801 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:52.988806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:52.988987 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:52.989133 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:52.989895 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:52.991256 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:52.992695 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:52.993669 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:52.993818 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:52.997140 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:52.997158 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:52.997179 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:52.997223 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:53.001222 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.001995 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.002007 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.002014 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.002171 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.002627 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.002635 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.002641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.003185 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:53.003404 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:53.003798 (+ 394us) server_negotiation.cc:299] Negotiation successful
0808 13:02:53.004110 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":456,"threads_started":1}
I20260808 13:02:53.007431 27529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } has no permanent_uuid. Determining permanent_uuid...
I20260808 13:02:53.008836 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.008279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40012)
0808 13:02:53.008484 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.008637 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 13:02:53.009119 27466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
I20260808 13:02:53.011065 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.010602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40018)
0808 13:02:53.010804 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.010922 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 13:02:53.011315 27529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:39503: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.167.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.60.keytab.
Entry for principal kudu/127.24.167.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.60.keytab.
I20260808 13:02:53.031847 25244 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
/tmp/dist-test-taskMF3Yup/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.60:39503
--webserver_interface=127.24.167.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.60.keytab
--principal=kudu/127.24.167.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.167.20:41167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.167.62:46349,127.24.167.61:40567,127.24.167.60:39503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 13:02:53.060678 27529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } attempt: 1
I20260808 13:02:53.062032 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.061367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40020)
0808 13:02:53.061634 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.061801 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 13:02:53.063311 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.062803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40034)
0808 13:02:53.063006 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.063127 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 13:02:53.063545 27529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:39503: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
I20260808 13:02:53.111819 27433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } attempt: 5
Aug 08 13:02:53 dist-test-slave-tk9z krb5kdc[27332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194172, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for kudu/127.24.167.61@KRBTEST.COM
I20260808 13:02:53.126881 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.112717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40044 (local address 127.24.167.61:40567)
0808 13:02:53.112933 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.112937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.112964 (+ 27us) server_negotiation.cc:407] Connection header received
0808 13:02:53.113162 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.113166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.113248 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.113332 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:53.114435 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.115132 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.116833 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.117277 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.117372 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.121375 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:53.121381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:53.121384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:53.121420 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:53.124160 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.124856 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.124862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.124866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.125005 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.125391 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.125396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.125400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.125703 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:53.125870 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:53.126262 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 13:02:53.126534 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260808 13:02:53.126888 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.112614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:40567 (local address 127.0.0.1:40044)
0808 13:02:53.112860 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.112885 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:53.113028 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:53.113408 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:53.113413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:53.113954 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:53.114226 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:53.114244 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.115274 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.115279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:53.116669 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:53.116686 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.117430 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.117436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:53.117697 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.118448 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:53.118479 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:53.121156 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:53.124319 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:53.124324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:53.124329 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:53.124732 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:53.125123 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:53.125129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:53.125133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:53.125279 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:53.125827 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:53.125831 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:53.126003 (+ 172us) client_negotiation.cc:769] Sending connection context
0808 13:02:53.126155 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 13:02:53.126543 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260808 13:02:53.128041 27433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } has no permanent_uuid. Determining permanent_uuid...
I20260808 13:02:53.129212 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.128802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40044)
0808 13:02:53.128992 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.129070 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 13:02:53.130223 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.129871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40052)
0808 13:02:53.130040 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.130121 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 13:02:53.130491 27433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:39503: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
I20260808 13:02:53.131781 27529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } attempt: 2
I20260808 13:02:53.132731 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.132273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40068)
0808 13:02:53.132459 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.132560 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 13:02:53.133761 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.133386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40076)
0808 13:02:53.133536 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.133623 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 13:02:53.133975 27529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:39503: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
I20260808 13:02:53.181793 27433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } attempt: 1
I20260808 13:02:53.183097 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.182609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40082)
0808 13:02:53.182809 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.182907 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 13:02:53.184106 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.183775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40092)
0808 13:02:53.183923 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.183998 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 13:02:53.184341 27433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:39503: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
I20260808 13:02:53.225225 27529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } attempt: 3
I20260808 13:02:53.226583 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.225986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40098)
0808 13:02:53.226239 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.226365 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 13:02:53.227708 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.227220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40100)
0808 13:02:53.227462 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.227557 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260808 13:02:53.227963 27529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:39503: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
I20260808 13:02:53.252588 27433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } attempt: 2
I20260808 13:02:53.253901 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.253386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40106)
0808 13:02:53.253630 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.253725 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 13:02:53.255100 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.254713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40122)
0808 13:02:53.254913 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.254991 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260808 13:02:53.255317 27433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:39503: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
I20260808 13:02:53.361544 27433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } attempt: 3
I20260808 13:02:53.362911 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.362293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40126)
0808 13:02:53.362550 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.362662 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260808 13:02:53.363983 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.363595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40140)
0808 13:02:53.363812 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.363883 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260808 13:02:53.364187 27433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:39503: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
I20260808 13:02:53.371201 27529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } attempt: 4
I20260808 13:02:53.372339 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.371804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40142)
0808 13:02:53.372046 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.372159 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 13:02:53.373407 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.373040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40146)
0808 13:02:53.373193 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.373280 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 13:02:53.373625 27529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.167.60:39503: Network error: Client connection negotiation failed: client connection to 127.24.167.60:39503: connect: Connection refused (error 111)
W20260808 13:02:53.395888 27539 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 13:02:53.396234 27539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 13:02:53.396356 27539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 13:02:53.396502 27539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 13:02:53.403955 27539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 13:02:53.404057 27539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 13:02:53.404150 27539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 13:02:53.404230 27539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 13:02:53.404336 27539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 13:02:53.414654 27539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.167.20:41167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.167.62:46349,127.24.167.61:40567,127.24.167.60:39503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/krb5kdc/kudu_127.24.167.60.keytab
--principal=kudu/127.24.167.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.167.60:39503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.167.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 12:05:15 UTC on ccycloud.quasar-wzjnwn.root.comops.site
build id 4574795
ASAN enabled
I20260808 13:02:53.415901 27539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 13:02:53.417229 27539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 13:02:53.429117 27546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:53.429071 27547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 13:02:53.429359 27550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 13:02:53.430069 27539 server_base.cc:1034] running on GCE node
I20260808 13:02:53.430943 27539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 13:02:53.432399 27539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 13:02:53.433626 27539 hybrid_clock.cc:648] HybridClock initialized: now 1786194173433555 us; error 45 us; skew 500 ppm
I20260808 13:02:53.433971 27539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 13:02:53 dist-test-slave-tk9z krb5kdc[27332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194173, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 13:02:53.437181 27539 init.cc:377] Logged in from keytab as kudu/127.24.167.60@KRBTEST.COM (short username kudu)
I20260808 13:02:53.438760 27539 webserver.cc:460] Webserver started at http://127.24.167.60:36937/ using document root <none> and password file <none>
I20260808 13:02:53.439463 27539 fs_manager.cc:359] Metadata directory not provided
I20260808 13:02:53.439592 27539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 13:02:53.439921 27539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 13:02:53.442939 27539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/data/instance:
uuid: "46065f32ff1c4e6cac0c63545a7f197d"
format_stamp: "Formatted at 2026-08-08 13:02:53 on dist-test-slave-tk9z"
server_key: "ee3300b2d82ee1064d3e649843c08b93"
server_key_iv: "8e813485262fe09125db0f7531bb2187"
server_key_version: "encryptionkey@0"
I20260808 13:02:53.443749 27539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/wal/instance:
uuid: "46065f32ff1c4e6cac0c63545a7f197d"
format_stamp: "Formatted at 2026-08-08 13:02:53 on dist-test-slave-tk9z"
server_key: "ee3300b2d82ee1064d3e649843c08b93"
server_key_iv: "8e813485262fe09125db0f7531bb2187"
server_key_version: "encryptionkey@0"
I20260808 13:02:53.448822 27539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 13:02:53.452325 27559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 13:02:53.453756 27539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 13:02:53.453950 27539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/data,/tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/wal
uuid: "46065f32ff1c4e6cac0c63545a7f197d"
format_stamp: "Formatted at 2026-08-08 13:02:53 on dist-test-slave-tk9z"
server_key: "ee3300b2d82ee1064d3e649843c08b93"
server_key_iv: "8e813485262fe09125db0f7531bb2187"
server_key_version: "encryptionkey@0"
I20260808 13:02:53.454182 27539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 13:02:53.481710 27539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 13:02:53.486050 27539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 13:02:53.486334 27539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 13:02:53.504907 27539 rpc_server.cc:307] RPC server started. Bound to: 127.24.167.60:39503
I20260808 13:02:53.504930 27631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.167.60:39503 every 8 connection(s)
I20260808 13:02:53.506287 27539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/data/info.pb
I20260808 13:02:53.507473 25244 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu as pid 27539
I20260808 13:02:53.507704 25244 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMF3Yup/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786193784050718-25244-0/minicluster-data/master-2/wal/instance
I20260808 13:02:53.508242 25244 external_mini_cluster.cc:1428] Setting key c4192a98f204cb2c67144eb269eaa1b9
I20260808 13:02:53.511041 27632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 13:02:53.519477 27632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 13:02:53 dist-test-slave-tk9z krb5kdc[27332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.60@KRBTEST.COM
Aug 08 13:02:53 dist-test-slave-tk9z krb5kdc[27332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194173, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.60@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
I20260808 13:02:53.532289 27635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.510262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40152 (local address 127.24.167.60:39503)
0808 13:02:53.511165 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.511184 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.511225 (+ 41us) server_negotiation.cc:407] Connection header received
0808 13:02:53.512850 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.512873 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.513221 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.513575 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:53.515340 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.516699 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.518532 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.519175 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.519306 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.524499 (+ 5193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:53.524507 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:53.524516 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:53.524553 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:53.527891 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.528725 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.528732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.528737 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.528895 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.529336 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.529342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.529347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.529913 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:53.530164 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:53.530681 (+ 517us) server_negotiation.cc:299] Negotiation successful
0808 13:02:53.531096 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":385,"threads_started":1}
I20260808 13:02:53.539709 27637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.522714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.62:46349 (local address 127.0.0.1:34712)
0808 13:02:53.523609 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.523651 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:53.523816 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:53.524207 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:53.524215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:53.524910 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:53.525247 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:53.525265 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.526730 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.526734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:53.527852 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:53.527865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.528736 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.528741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:53.528870 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.529528 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:53.529600 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:53.532151 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:53.537171 (+ 5020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:53.537180 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:53.537186 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:53.537770 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:53.538140 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:53.538147 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:53.538151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:53.538307 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:53.538879 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:53.538885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:53.539048 (+ 163us) client_negotiation.cc:769] Sending connection context
0808 13:02:53.539190 (+ 142us) client_negotiation.cc:240] Negotiation successful
0808 13:02:53.539420 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"spinlock_wait_cycles":12672,"thread_start_us":491,"threads_started":1}
I20260808 13:02:53.539976 27636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.522714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.24.167.62:46349)
0808 13:02:53.523247 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.523254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.523738 (+ 484us) server_negotiation.cc:407] Connection header received
0808 13:02:53.523950 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.523961 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.524026 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.524142 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:53.525415 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.526569 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.528007 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.528433 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.528529 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.532363 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:53.532370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:53.532375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:53.532430 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:53.537042 (+ 4612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.537885 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.537889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.537891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.538028 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.538455 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.538459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.538461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.538738 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:53.538928 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:53.539348 (+ 420us) server_negotiation.cc:299] Negotiation successful
0808 13:02:53.539570 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":254,"threads_started":1}
I20260808 13:02:53.541358 27632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } has no permanent_uuid. Determining permanent_uuid...
I20260808 13:02:53.541378 27433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } attempt: 4
Aug 08 13:02:53 dist-test-slave-tk9z krb5kdc[27332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.62@KRBTEST.COM
Aug 08 13:02:53 dist-test-slave-tk9z krb5kdc[27332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194172, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.62@KRBTEST.COM for kudu/127.24.167.60@KRBTEST.COM
Aug 08 13:02:53 dist-test-slave-tk9z krb5kdc[27332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194173, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.60@KRBTEST.COM for kudu/127.24.167.61@KRBTEST.COM
I20260808 13:02:53.554060 27638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.535063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.24.167.62:46349)
0808 13:02:53.535640 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.535644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.535664 (+ 20us) server_negotiation.cc:407] Connection header received
0808 13:02:53.535708 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.535712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.535765 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.535854 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:53.537481 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.538664 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.540371 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.541036 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.541164 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.547069 (+ 5905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:53.547075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:53.547080 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:53.547118 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:53.550799 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.551540 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.551546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.551549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.551684 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.552157 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.552162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.552170 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.552548 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:53.552834 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:53.553172 (+ 338us) server_negotiation.cc:299] Negotiation successful
0808 13:02:53.553407 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":263,"threads_started":1}
I20260808 13:02:53.557403 27635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.542242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.24.167.60:39503)
0808 13:02:53.542417 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.542423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.542439 (+ 16us) server_negotiation.cc:407] Connection header received
0808 13:02:53.542486 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.542490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.542554 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.542704 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:53.543370 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.544149 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.545721 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.546406 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.546594 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.550642 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:53.550648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:53.550656 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:53.550699 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:53.553852 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.554687 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.554692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.554696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.554824 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.555277 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.555282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.555286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.555631 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:53.555761 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:53.556670 (+ 909us) server_negotiation.cc:299] Negotiation successful
0808 13:02:53.556988 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":48}
I20260808 13:02:53.557642 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.541985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40162)
0808 13:02:53.542187 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.542207 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:53.542309 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:53.542744 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:53.542750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:53.543096 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:53.543234 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:53.543242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.544293 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.544300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:53.545581 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:53.545593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.546625 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.546633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:53.546829 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.547444 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:53.547461 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:53.550458 (+ 2997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:53.554001 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:53.554006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:53.554010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:53.554567 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:53.554933 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:53.554940 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:53.554946 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:53.555133 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:53.556120 (+ 987us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:53.556125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:53.556283 (+ 158us) client_negotiation.cc:769] Sending connection context
0808 13:02:53.556603 (+ 320us) client_negotiation.cc:240] Negotiation successful
0808 13:02:53.557044 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 13:02:53.558099 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.542579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.24.167.61:40567)
0808 13:02:53.542778 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.542784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.542801 (+ 17us) server_negotiation.cc:407] Connection header received
0808 13:02:53.542861 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.542865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.542920 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.543025 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:53.543946 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.544724 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.546517 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.547083 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.547225 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.551313 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:53.551320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:53.551325 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:53.551365 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:53.554983 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.555756 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.555761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.555764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.555941 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.556386 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.556391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.556394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.556759 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:53.556931 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:53.557436 (+ 505us) server_negotiation.cc:299] Negotiation successful
0808 13:02:53.557668 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 13:02:53.559062 27637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.542048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.61:40567 (local address 127.0.0.1:40054)
0808 13:02:53.542565 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.542584 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:53.542675 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:53.543046 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:53.543050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:53.543548 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:53.543768 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:53.543783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.544862 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.544869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:53.546344 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:53.546362 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.547275 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.547284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:53.548269 (+ 985us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.548799 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:53.548816 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:53.551104 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:53.555110 (+ 4006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:53.555115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:53.555118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:53.555623 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:53.556062 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:53.556067 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:53.556070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:53.556241 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:53.556989 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:53.556993 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:53.557145 (+ 152us) client_negotiation.cc:769] Sending connection context
0808 13:02:53.558421 (+ 1276us) client_negotiation.cc:240] Negotiation successful
0808 13:02:53.558692 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":47}
I20260808 13:02:53.560174 27632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } has no permanent_uuid. Determining permanent_uuid...
I20260808 13:02:53.566648 27433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265: Bootstrap starting.
Aug 08 13:02:53 dist-test-slave-tk9z krb5kdc[27332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.167.61@KRBTEST.COM
Aug 08 13:02:53 dist-test-slave-tk9z krb5kdc[27332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194173, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.60@KRBTEST.COM for kudu/127.24.167.60@KRBTEST.COM
I20260808 13:02:53.572631 27433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:53.574052 27433 log.cc:824] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:53.577606 27637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.560933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40164)
0808 13:02:53.561221 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.561237 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:53.561325 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:53.562135 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:53.562139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:53.562616 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:53.562779 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:53.562790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.564099 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.564105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:53.565477 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:53.565496 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.566746 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.566752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:53.567073 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.567684 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:53.567711 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:53.571084 (+ 3373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:53.575043 (+ 3959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:53.575048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:53.575052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:53.575398 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:53.575753 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:53.575758 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:53.575762 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:53.575925 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:53.576632 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:53.576635 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:53.576794 (+ 159us) client_negotiation.cc:769] Sending connection context
0808 13:02:53.576951 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 13:02:53.577260 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260808 13:02:53.578088 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.556233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40060 (local address 127.24.167.61:40567)
0808 13:02:53.558434 (+ 2201us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.558440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.558456 (+ 16us) server_negotiation.cc:407] Connection header received
0808 13:02:53.558502 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.558507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.558571 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.558664 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:53.560936 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.561810 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.563540 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.564201 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.564286 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.570766 (+ 6480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:53.570774 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:53.570778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:53.570823 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:53.574891 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.575858 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.575863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.575871 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.576039 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.576501 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.576506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.576510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.576849 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:53.577125 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:53.577500 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 13:02:53.577732 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2056,"thread_start_us":16091,"threads_started":1}
I20260808 13:02:53.578152 27433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265: No bootstrap required, opened a new log
I20260808 13:02:53.578555 27635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.561235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.24.167.60:39503)
0808 13:02:53.561829 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.561838 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.561859 (+ 21us) server_negotiation.cc:407] Connection header received
0808 13:02:53.561908 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.561913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.561983 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.562090 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:53.562930 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.563958 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.565642 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.566336 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.566522 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.571293 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:53.571299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:53.571304 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:53.571339 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:53.574891 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.575515 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.575522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.575526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.575652 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.576050 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.576054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.576056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.576404 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:53.576581 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:53.577986 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0808 13:02:53.578191 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473}
I20260808 13:02:53.583752 25244 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 13:02:53.584086 27632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d: Bootstrap starting.
I20260808 13:02:53.587245 27632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:53.587356 27403 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9f7dee8229a34e01ae6601c581475265"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34718
I20260808 13:02:53.588591 27632 log.cc:824] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:53.588909 27433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } }
I20260808 13:02:53.589262 27433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7dee8229a34e01ae6601c581475265, State: Initialized, Role: FOLLOWER
I20260808 13:02:53.589769 27433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } }
I20260808 13:02:53.590767 27644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } } }
I20260808 13:02:53.591243 27644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:02:53.592052 27632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d: No bootstrap required, opened a new log
I20260808 13:02:53.592218 27433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:53.592342 27403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 13:02:53.592576 27403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:53.597560 27403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } }
I20260808 13:02:53.600728 27403 leader_election.cc:288] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [CANDIDATE]: Term 1 election: Requested vote from peers 8200805955f1414a8976796841bdedcf (127.24.167.61:40567), 46065f32ff1c4e6cac0c63545a7f197d (127.24.167.60:39503)
I20260808 13:02:53.600627 27632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } }
I20260808 13:02:53.601074 27632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46065f32ff1c4e6cac0c63545a7f197d, State: Initialized, Role: FOLLOWER
I20260808 13:02:53.601599 27632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } }
I20260808 13:02:53.603207 27656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } } }
I20260808 13:02:53.604164 27632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:53.604257 27597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f7dee8229a34e01ae6601c581475265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "46065f32ff1c4e6cac0c63545a7f197d"
I20260808 13:02:53.604730 27597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:53.609579 27597 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7dee8229a34e01ae6601c581475265 in term 1.
I20260808 13:02:53.610571 27656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:02:53.610982 27371 leader_election.cc:302] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46065f32ff1c4e6cac0c63545a7f197d, 9f7dee8229a34e01ae6601c581475265; no voters:
I20260808 13:02:53.611649 27644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 13:02:53.612500 27644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [term 1 LEADER]: Becoming Leader. State: Replica: 9f7dee8229a34e01ae6601c581475265, State: Running, Role: LEADER
I20260808 13:02:53.613240 27644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } }
I20260808 13:02:53.628206 27661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f7dee8229a34e01ae6601c581475265. Latest consensus state: current_term: 1 leader_uuid: "9f7dee8229a34e01ae6601c581475265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } } }
I20260808 13:02:53.628865 27661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [sys.catalog]: This master's current role is: LEADER
W20260808 13:02:53.640738 27374 tablet.cc:2369] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 13:02:53.646960 27677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 13:02:53.660691 27694 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 13:02:53.663339 27677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 13:02:53.672816 27529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } attempt: 5
I20260808 13:02:53.663980 27539 master_runner.cc:186] Error getting master registration for 127.24.167.62:46349: OK, instance_id { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" instance_seqno: 1786194172402005 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 13:02:53.674705 27539 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 13:02:53.681746 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.640797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.24.167.61:40567)
0808 13:02:53.641021 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.641027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.641799 (+ 772us) server_negotiation.cc:407] Connection header received
0808 13:02:53.642072 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.642081 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.642198 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.642336 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:53.645400 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.646859 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.652040 (+ 5181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.653007 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.653148 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.671922 (+ 18774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:53.671932 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:53.671937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:53.671988 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:53.676532 (+ 4544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.677546 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.677554 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.677559 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.677870 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.678554 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.678561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.678564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.679177 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:53.680610 (+ 1433us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:53.680830 (+ 220us) server_negotiation.cc:299] Negotiation successful
0808 13:02:53.681186 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 13:02:53.693849 27635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.650921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.24.167.60:39503)
0808 13:02:53.651187 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.651196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.675035 (+ 23839us) server_negotiation.cc:407] Connection header received
0808 13:02:53.675212 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.675217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.675381 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.675522 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:53.679013 (+ 3491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.680102 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.682063 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.682776 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.682914 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.686245 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:53.686253 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:53.686258 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:53.686299 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:53.689923 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.690916 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.690922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.690926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.691078 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.691569 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.691575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.691581 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.691943 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:53.692195 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:53.693220 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0808 13:02:53.693471 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 13:02:53.703547 27638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.640306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34724 (local address 127.24.167.62:46349)
0808 13:02:53.640550 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.640558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.674491 (+ 33933us) server_negotiation.cc:407] Connection header received
0808 13:02:53.676729 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.676736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.676850 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.676974 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:53.684641 (+ 7667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.686697 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.688456 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.689176 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.689296 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.692709 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:53.692715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:53.692720 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:53.692765 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:53.699371 (+ 6606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.700394 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.700400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.700404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.700566 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.701044 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.701050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.701053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.701474 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:53.701746 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:53.702886 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0808 13:02:53.703153 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 13:02:53.706220 27597 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d [term 1 FOLLOWER]: Refusing update from remote peer 9f7dee8229a34e01ae6601c581475265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:02:53.707212 27661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Aug 08 13:02:53 dist-test-slave-tk9z krb5kdc[27332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786194172, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.167.61@KRBTEST.COM for kudu/127.24.167.60@KRBTEST.COM
I20260808 13:02:53.690207 27348 master_runner.cc:186] Error getting master registration for 127.24.167.61:40567: OK, instance_id { permanent_uuid: "8200805955f1414a8976796841bdedcf" instance_seqno: 1786194172941245 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 13:02:53.710834 27348 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 13:02:53.719033 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.673436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.167.60:39503 (local address 127.0.0.1:40182)
0808 13:02:53.673726 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0808 13:02:53.673746 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 13:02:53.673857 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 13:02:53.694618 (+ 20761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 13:02:53.694624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 13:02:53.695147 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0808 13:02:53.695377 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:53.695391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.696887 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.696893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:53.698296 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 13:02:53.698314 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.707436 (+ 9122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.707444 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 13:02:53.707703 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.708385 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 13:02:53.708405 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 13:02:53.711139 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 13:02:53.716052 (+ 4913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:53.716057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:53.716061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:53.716608 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:53.716991 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 13:02:53.716998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 13:02:53.717003 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 13:02:53.717204 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 13:02:53.717914 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 13:02:53.717920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 13:02:53.718092 (+ 172us) client_negotiation.cc:769] Sending connection context
0808 13:02:53.718268 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 13:02:53.718628 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260808 13:02:53.719725 27635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.673547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.24.167.60:39503)
0808 13:02:53.694238 (+ 20691us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.694243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.694261 (+ 18us) server_negotiation.cc:407] Connection header received
0808 13:02:53.694308 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.694314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.694420 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.695546 (+ 1126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:53.695690 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.696724 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.706538 (+ 9814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.707261 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.707390 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.711330 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 13:02:53.711340 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 13:02:53.711344 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 13:02:53.711382 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 13:02:53.715882 (+ 4500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.716760 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.716764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.716766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.716879 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 13:02:53.717347 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 13:02:53.717351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 13:02:53.717354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 13:02:53.717774 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 13:02:53.717962 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:53.719052 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0808 13:02:53.719339 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20582,"thread_start_us":26232,"threads_started":1}
I20260808 13:02:53.720362 27656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f7dee8229a34e01ae6601c581475265. Latest consensus state: current_term: 1 leader_uuid: "9f7dee8229a34e01ae6601c581475265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } } }
I20260808 13:02:53.720609 27656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:02:53.724107 27699 mvcc.cc:205] Tried to move back new op lower bound from 7316251335483695104 to 7316251335128064000. Current Snapshot: MvccSnapshot[applied={T|T < 7316251335483695104}]
I20260808 13:02:53.726284 27661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f7dee8229a34e01ae6601c581475265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } } }
I20260808 13:02:53.726645 27661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:53.729089 27677 catalog_manager.cc:1347] Generated new cluster ID: 48751aa0e1734c3a91a8a84999dcc6d3
I20260808 13:02:53.729210 27677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 13:02:53.729197 27529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf: Bootstrap starting.
I20260808 13:02:53.733052 27529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf: Neither blocks nor log segments found. Creating new log.
I20260808 13:02:53.734187 27529 log.cc:824] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf: Log is configured to *not* fsync() on all Append() calls
I20260808 13:02:53.734882 27700 mvcc.cc:205] Tried to move back new op lower bound from 7316251335483695104 to 7316251335128064000. Current Snapshot: MvccSnapshot[applied={T|T < 7316251335483695104}]
I20260808 13:02:53.738147 27529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf: No bootstrap required, opened a new log
I20260808 13:02:53.751793 27529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } }
I20260808 13:02:53.752732 27529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8200805955f1414a8976796841bdedcf, State: Initialized, Role: FOLLOWER
I20260808 13:02:53.752642 27656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f7dee8229a34e01ae6601c581475265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } } }
I20260808 13:02:53.752921 27656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:02:53.753427 27529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } }
I20260808 13:02:53.754930 27706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } } }
I20260808 13:02:53.755962 27706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:02:53.758219 27529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf [sys.catalog]: configured and running, proceeding with master startup.
I20260808 13:02:53.758755 27498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf [term 0 FOLLOWER]: Advancing to term 1
I20260808 13:02:53.762604 27501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f7dee8229a34e01ae6601c581475265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8200805955f1414a8976796841bdedcf"
I20260808 13:02:53.764065 27498 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf [term 1 FOLLOWER]: Refusing update from remote peer 9f7dee8229a34e01ae6601c581475265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 13:02:53.769208 27501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7dee8229a34e01ae6601c581475265 in term 1.
I20260808 13:02:53.771845 27661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 13:02:53.799590 27677 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 13:02:53.801923 27677 catalog_manager.cc:1504] Loading token signing keys...
I20260808 13:02:53.815414 27644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f7dee8229a34e01ae6601c581475265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } } }
I20260808 13:02:53.815781 27644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265 [sys.catalog]: This master's current role is: LEADER
I20260808 13:02:53.824013 27706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f7dee8229a34e01ae6601c581475265. Latest consensus state: current_term: 1 leader_uuid: "9f7dee8229a34e01ae6601c581475265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } } }
I20260808 13:02:53.824532 27706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:02:53.825625 27715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f7dee8229a34e01ae6601c581475265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7dee8229a34e01ae6601c581475265" member_type: VOTER last_known_addr { host: "127.24.167.62" port: 46349 } } peers { permanent_uuid: "8200805955f1414a8976796841bdedcf" member_type: VOTER last_known_addr { host: "127.24.167.61" port: 40567 } } peers { permanent_uuid: "46065f32ff1c4e6cac0c63545a7f197d" member_type: VOTER last_known_addr { host: "127.24.167.60" port: 39503 } } }
I20260808 13:02:53.826188 27715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf [sys.catalog]: This master's current role is: FOLLOWER
I20260808 13:02:53.839208 27677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f7dee8229a34e01ae6601c581475265: Generated new TSK 0
I20260808 13:02:53.840859 27677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 13:02:53.858063 27726 catalog_manager.cc:1259] Loaded cluster ID: 48751aa0e1734c3a91a8a84999dcc6d3
I20260808 13:02:53.858182 27726 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf: loading cluster ID for follower catalog manager: success
I20260808 13:02:53.864866 27726 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf: acquiring CA information for follower catalog manager: success
I20260808 13:02:53.867741 27726 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8200805955f1414a8976796841bdedcf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 13:02:53.953351 27638 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:53.940109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34740 (local address 127.24.167.62:46349)
0808 13:02:53.940314 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.940321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.951086 (+ 10765us) server_negotiation.cc:407] Connection header received
0808 13:02:53.951405 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.951415 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.951514 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.951604 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:53.953034 (+ 1430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34740: BlockingRecv error: recv got EOF from 127.0.0.1:34740 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 13:02:53.955334 27635 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:53.940161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40190 (local address 127.24.167.60:39503)
0808 13:02:53.940377 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.940385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.952766 (+ 12381us) server_negotiation.cc:407] Connection header received
0808 13:02:53.953285 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.953293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.953392 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.953481 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:53.955060 (+ 1579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40190: BlockingRecv error: recv got EOF from 127.0.0.1:40190 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 13:02:53.959280 27536 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:53.950589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.24.167.61:40567)
0808 13:02:53.950908 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.950917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.957069 (+ 6152us) server_negotiation.cc:407] Connection header received
0808 13:02:53.957234 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.957242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.957335 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.957489 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:53.959044 (+ 1555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40078: BlockingRecv error: recv got EOF from 127.0.0.1:40078 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
I20260808 13:02:53.994930 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:53.984013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.24.167.61:40567)
0808 13:02:53.984233 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:53.984240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:53.986225 (+ 1985us) server_negotiation.cc:407] Connection header received
0808 13:02:53.986579 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:53.986586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:53.986819 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:53.987022 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:53.987613 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.988949 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.991294 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:53.992120 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:53.992340 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:53.993082 (+ 742us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:53.994000 (+ 918us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:53.994180 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:53.994300 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 13:02:53.994471 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260808 13:02:54.014066 27635 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:54.011190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.24.167.60:39503)
0808 13:02:54.011418 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.011426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.012169 (+ 743us) server_negotiation.cc:407] Connection header received
0808 13:02:54.012469 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.012476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.012621 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.012748 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:54.013755 (+ 1007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40200: BlockingRecv error: recv got EOF from 127.0.0.1:40200 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 13:02:54.033545 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:54.020509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40094 (local address 127.24.167.61:40567)
0808 13:02:54.020727 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.020736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.026356 (+ 5620us) server_negotiation.cc:407] Connection header received
0808 13:02:54.026623 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.026631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.026758 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.026913 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:54.027447 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.028743 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.031131 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.031919 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.032069 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:54.032609 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:54.032795 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:54.032907 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:54.033088 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 13:02:54.033193 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260808 13:02:54.052832 27635 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:54.042144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.24.167.60:39503)
0808 13:02:54.042340 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.042348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.051086 (+ 8738us) server_negotiation.cc:407] Connection header received
0808 13:02:54.051298 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.051304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.051421 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.051541 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:54.052576 (+ 1035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40216: BlockingRecv error: recv got EOF from 127.0.0.1:40216 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 13:02:54.066843 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:54.058824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.24.167.61:40567)
0808 13:02:54.059027 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.059033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.059863 (+ 830us) server_negotiation.cc:407] Connection header received
0808 13:02:54.060089 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.060096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.060194 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.060306 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:54.060823 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.062152 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.064370 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.065111 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.065260 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:54.065812 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:54.065999 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:54.066102 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:54.066287 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 13:02:54.066453 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 13:02:54.083588 27635 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:54.080934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.24.167.60:39503)
0808 13:02:54.081136 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.081143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.081754 (+ 611us) server_negotiation.cc:407] Connection header received
0808 13:02:54.081981 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.081988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.082093 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.082210 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:54.083306 (+ 1096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40232: BlockingRecv error: recv got EOF from 127.0.0.1:40232 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 13:02:54.108517 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:54.100324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.24.167.61:40567)
0808 13:02:54.100605 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.100612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.101462 (+ 850us) server_negotiation.cc:407] Connection header received
0808 13:02:54.101743 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.101756 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.101892 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.101996 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:54.102598 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.104026 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.106141 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.106980 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.107138 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:54.107633 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:54.107784 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:54.107889 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:54.108069 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 13:02:54.108168 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
W20260808 13:02:54.119329 27635 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:54.116621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.24.167.60:39503)
0808 13:02:54.116949 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.116957 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.117618 (+ 661us) server_negotiation.cc:407] Connection header received
0808 13:02:54.117845 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.117852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.117962 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.118086 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:54.119097 (+ 1011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40242: BlockingRecv error: recv got EOF from 127.0.0.1:40242 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
I20260808 13:02:54.139128 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:54.131395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40138 (local address 127.24.167.61:40567)
0808 13:02:54.131605 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.131613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.132206 (+ 593us) server_negotiation.cc:407] Connection header received
0808 13:02:54.132509 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.132515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.132629 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.132757 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:54.133279 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.134602 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.136712 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.137457 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.137613 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:54.138197 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:54.138401 (+ 204us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:54.138501 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:54.138680 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 13:02:54.138773 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260808 13:02:54.152693 27635 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:54.146824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.24.167.60:39503)
0808 13:02:54.147024 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.147031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.151021 (+ 3990us) server_negotiation.cc:407] Connection header received
0808 13:02:54.151210 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.151216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.151321 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.151445 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:54.152398 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40250: BlockingRecv error: recv got EOF from 127.0.0.1:40250 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 13:02:54.182221 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:54.173855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40146 (local address 127.24.167.61:40567)
0808 13:02:54.174093 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.174102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.174811 (+ 709us) server_negotiation.cc:407] Connection header received
0808 13:02:54.175070 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.175077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.175188 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.175312 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:54.175880 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.177239 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.179553 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.180245 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.180393 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:54.180985 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:54.181192 (+ 207us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:54.181311 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:54.181493 (+ 182us) server_negotiation.cc:299] Negotiation successful
0808 13:02:54.181603 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260808 13:02:54.195279 27635 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:54.192577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.24.167.60:39503)
0808 13:02:54.192781 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.192787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.193363 (+ 576us) server_negotiation.cc:407] Connection header received
0808 13:02:54.193668 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.193675 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.193834 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.193955 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:54.194980 (+ 1025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40256: BlockingRecv error: recv got EOF from 127.0.0.1:40256 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 13:02:54.240083 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:54.231777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.24.167.61:40567)
0808 13:02:54.232027 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.232037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.232689 (+ 652us) server_negotiation.cc:407] Connection header received
0808 13:02:54.232935 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.232941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.233051 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.233171 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:54.233780 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.235183 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.237564 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.238335 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.238511 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:54.239063 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:54.239221 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:54.239336 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:54.239542 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 13:02:54.239649 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260808 13:02:54.250478 27635 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:54.247859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.24.167.60:39503)
0808 13:02:54.248071 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.248078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.248694 (+ 616us) server_negotiation.cc:407] Connection header received
0808 13:02:54.248907 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.248913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.249021 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.249142 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:54.250129 (+ 987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40266: BlockingRecv error: recv got EOF from 127.0.0.1:40266 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 13:02:54.324982 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:54.318342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.24.167.61:40567)
0808 13:02:54.318592 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.318600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.319059 (+ 459us) server_negotiation.cc:407] Connection header received
0808 13:02:54.319239 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.319245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.319365 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.319477 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:54.319909 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.321223 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.322743 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.323574 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.323715 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:54.324096 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:54.324262 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:54.324369 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:54.324513 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 13:02:54.324617 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260808 13:02:54.337531 27635 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:54.335220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.24.167.60:39503)
0808 13:02:54.335486 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.335492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.335853 (+ 361us) server_negotiation.cc:407] Connection header received
0808 13:02:54.336046 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.336050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.336130 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.336230 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:54.337235 (+ 1005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40280: BlockingRecv error: recv got EOF from 127.0.0.1:40280 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 13:02:54.474265 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:54.469222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.24.167.61:40567)
0808 13:02:54.469436 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.469442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.469748 (+ 306us) server_negotiation.cc:407] Connection header received
0808 13:02:54.469924 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.469929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.470008 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.470110 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:54.470597 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.471398 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.472702 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.473146 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.473247 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:54.473663 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:54.473781 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:54.473867 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:54.473984 (+ 117us) server_negotiation.cc:299] Negotiation successful
0808 13:02:54.474048 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260808 13:02:54.482260 27635 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 13:02:54.480424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40294 (local address 127.24.167.60:39503)
0808 13:02:54.480641 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.480649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.480852 (+ 203us) server_negotiation.cc:407] Connection header received
0808 13:02:54.480997 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.481001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.481071 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.481156 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 13:02:54.482052 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40294: BlockingRecv error: recv got EOF from 127.0.0.1:40294 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 13:02:54.665279 27694 catalog_manager.cc:1259] Loaded cluster ID: 48751aa0e1734c3a91a8a84999dcc6d3
I20260808 13:02:54.665400 27694 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d: loading cluster ID for follower catalog manager: success
I20260808 13:02:54.668103 27694 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d: acquiring CA information for follower catalog manager: success
I20260808 13:02:54.670246 27694 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 46065f32ff1c4e6cac0c63545a7f197d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 13:02:54.748602 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:54.742212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.24.167.61:40567)
0808 13:02:54.742473 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.742480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.742786 (+ 306us) server_negotiation.cc:407] Connection header received
0808 13:02:54.742977 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.742984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.743106 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.743233 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:54.743723 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.745016 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.746491 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.747202 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.747360 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:54.747784 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:54.747942 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:54.748047 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:54.748184 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 13:02:54.748282 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 13:02:54.762847 27635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:54.755270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40298 (local address 127.24.167.60:39503)
0808 13:02:54.755490 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.755497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.755785 (+ 288us) server_negotiation.cc:407] Connection header received
0808 13:02:54.755944 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.755951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.756228 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.756396 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:54.756806 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.758133 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.759791 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.760615 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.760754 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:54.761292 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:54.762057 (+ 765us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:54.762246 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:54.762310 (+ 64us) server_negotiation.cc:299] Negotiation successful
0808 13:02:54.762427 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 13:02:54.789774 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:54.769503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.24.167.61:40567)
0808 13:02:54.770511 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.770520 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.780899 (+ 10379us) server_negotiation.cc:407] Connection header received
0808 13:02:54.781251 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.781261 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.781391 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.781476 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:54.782506 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.784061 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.787457 (+ 3396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.788074 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.788214 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:54.788864 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:54.789010 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:54.789088 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:54.789350 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 13:02:54.789446 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822}
I20260808 13:02:54.793265 27876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:54.769532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.24.167.62:46349)
0808 13:02:54.785867 (+ 16335us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.785874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.785897 (+ 23us) server_negotiation.cc:407] Connection header received
0808 13:02:54.786021 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.786029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.786255 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.786527 (+ 272us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:54.787269 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.788436 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.790620 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.791155 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.791281 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:54.792004 (+ 723us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:54.792527 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:54.792666 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:54.792787 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 13:02:54.792883 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16195,"thread_start_us":17134,"threads_started":1}
I20260808 13:02:54.797333 27635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 13:02:54.769697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.24.167.60:39503)
0808 13:02:54.771345 (+ 1648us) server_negotiation.cc:206] Beginning negotiation
0808 13:02:54.771353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 13:02:54.788866 (+ 17513us) server_negotiation.cc:407] Connection header received
0808 13:02:54.789160 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 13:02:54.789167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 13:02:54.789272 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 13:02:54.789350 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 13:02:54.791368 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.792532 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.794821 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 13:02:54.795592 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 13:02:54.795702 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 13:02:54.796512 (+ 810us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 13:02:54.796647 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 13:02:54.796760 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 13:02:54.796895 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 13:02:54.796979 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1493}
I20260808 13:02:54.797871 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 27348
I20260808 13:02:54.798980 27426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:54.929752 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 27348
W20260808 13:02:54.958858 27466 connection.cc:535] server connection from 127.0.0.1:40044 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 13:02:54.959545 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 27444
I20260808 13:02:54.960737 27524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:55.072729 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 27444
I20260808 13:02:55.103626 25244 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu pid 27539
I20260808 13:02:55.104928 27624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 13:02:55.217581 25244 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMF3Yup/build/asan/bin/kudu with pid 27539
2026-08-08T13:02:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5021 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10072 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (391118 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST